Artificial intelligence psychological assessment method and system based on multimodal input

Abstract

The present invention discloses an artificial intelligence psychological assessment method and system based on multimodal input, the method comprising the following steps: S10: establishing and training a personal psychological model for psychological assessment; S20: receiving a psychological assessment request initiated by a user, and communicating with the user through the personal psychological model; S30: acquiring the multimodal data of the user during the dialogue with the user; S40: processing and analyzing the multimodal data of the user, and conducting a psychological health assessment on the user. The method can acquire and process various types of user data, such as text, voice, facial expressions, physiological signals, etc. This multimodal data fusion method can provide more comprehensive and accurate psychological state information, thereby improving the accuracy of psychological assessment. The method establishes and trains a personalized psychological model for each user, and can provide more targeted psychological assessment services.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an artificial intelligence psychological assessment method and system based on multimodal input. Background Art

[0002] With the accelerated pace of modern society and the intensified competition, more and more people are facing mental health problems. Traditional psychological assessment methods often rely on professional counselors or psychologists for face-to-face communication and treatment. This method is not only limited in resources, but also restricted by time, location and other factors, and it is difficult to meet the needs of the majority of users.

[0003] In recent years, the rapid development of artificial intelligence technology has provided new solutions for psychological assessment. By using technologies such as natural language processing, machine learning, and deep learning, artificial intelligence can simulate the thinking and behavior of human experts and conduct preliminary psychological assessments on users. However, most of the current artificial intelligence psychological assessment methods are based on single text or voice input, which makes it difficult to fully and accurately obtain information about the user's psychological state, thus affecting the accuracy of the assessment. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides an artificial intelligence psychological assessment method and system based on multimodal input to solve the above technical problems.

[0005] To achieve the above objectives, in a first aspect, an artificial intelligence psychological assessment method based on multimodal input is provided, the method comprising the following steps:

[0006] S10: Build and train personal psychological models for psychological assessment;

[0007] S20: receiving a psychological assessment request initiated by a user, and communicating with the user through the personal psychological model;

[0008] S30: Acquiring multimodal data of the user during the conversation with the user;

[0009] S40: Process and analyze the multimodal data of the user, and conduct a mental health assessment on the user.

[0010] In some possible implementations, step S10 specifically includes:

[0011] S11: Optimizing the pre-trained large language model, and adapting the large language model to the context of psychological assessment by stepwise distillation;

[0012] S12: Initialize the pre-trained large language model based on the user's basic information to obtain the first person psychological model;

[0013] S13: communicating with the user through the first personal psychological model;

[0014] S14: during the conversation, obtaining multimodal data of the user, wherein the multimodal data includes video data, audio signals, near infrared data, thermal imaging data, and high-speed photography data;

[0015] S15: Processing the multimodal data;

[0016] S16: Using transfer learning to input the processed multimodal data into the first personal psychological model for training iterative optimization, thereby obtaining a second personal psychological model.

[0017] In some possible implementations, step S12 specifically includes the following steps:

[0018] S121: Collect basic user information, and encode the basic user information into a processing format of a large language model, wherein the encoding method includes at least one of one-hot encoding, embedding vector, and combined features;

[0019] S122: Load a large pre-trained language model based on the Transformer architecture from storage;

[0020] S123: Adjusting parameters of the pre-trained large language model to obtain the first personal psychological model, which includes:

[0021] Add one or more new neural network layers on top of the pre-trained large language model neural network;

[0022] The encoded user basic information is used as the input of a pre-trained large language model with a new neural network layer added, combined with the input of each Transformer layer of the pre-trained large language model, and a set of adaptive weights is output;

[0023] Combining the basic weights and adaptive weights of the pre-trained large language model, a new weight matrix is ​​determined by a hybrid weighted summation method to personalize the pre-trained large language model to obtain the first personal psychological model; wherein the hybrid parameter α of the hybrid weighted summation has a value range between 0 and 1;

[0024] S124: Based on the user basic information, independently store the first personal psychological model corresponding to the user.

[0025] In some possible implementations, step S15 specifically includes step S151: processing the video data in the multimodal data;

[0026] Step S151 specifically includes: a video preprocessing step, a body movement data extraction step, a facial expression analysis step, and a pupil change detection step;

[0027] The video preprocessing step specifically includes: extracting frames from the video data to decompose the video data into separate video frames; adjusting the resolution of the video frames; normalizing the video data to standardize the brightness and contrast of the video frames to obtain preprocessed video frames;

[0028] The limb motion data extraction step specifically includes: using a human posture estimation algorithm to detect and track key points of the human body in the preprocessed video frame, and identifying specific limb motions and behavior patterns by analyzing the movement of key points of the human body over time, and generating limb motion data features;

[0029] The facial expression analysis step specifically includes: using a facial detection algorithm to identify the facial area in the preprocessed video frame, and using a facial feature point detection algorithm to track key feature points of the face to obtain facial expression data features of the current user;

[0030] Among them, the pupil change detection step specifically includes: using an eye area positioning algorithm to determine the points related to the eyes among the facial feature points in the preprocessed video frame, and using a pupil positioning algorithm to determine the exact position of the pupil, and then performing pupil size change analysis to generate pupil change data features.

[0031] In some possible implementations, step S15 specifically includes step S152: processing the audio signal in the multimodal data; step S152 specifically includes: an audio preprocessing sub-step, a speech recognition sub-step, a tone and intonation analysis sub-step, and a rhythm pause analysis sub-step;

[0032] The audio preprocessing sub-step comprises:

[0033] Performing noise reduction processing on the audio signal;

[0034] Segment the audio signal after the noise reduction process to obtain an audio frame after the noise reduction and segmentation process;

[0035] Extracting spectrum features from the audio frames that have undergone noise reduction and segmentation processing, and normalizing the extracted spectrum features;

[0036] The speech recognition sub-step comprises:

[0037] Taking the preprocessed audio signal as input, using a sequence-to-sequence model to perform speech recognition, and outputting a preliminary text sequence corresponding to the preprocessed audio signal; wherein the sequence-to-sequence model includes an encoder and a decoder with an attention mechanism, and introduces a reinforcement learning strategy, using the minimum word error rate as a reward function for model training;

[0038] Taking the preprocessed audio signal and the preliminary text sequence as input, performing joint training and joint decoding in combination with the converter model, and outputting an optimized text sequence;

[0039] The tone and intonation analysis sub-step includes:

[0040] Extracting intonation patterns from the preprocessed audio signal using a pitch detection algorithm;

[0041] Extracting key data features from the preprocessed audio signal using Mel frequency cepstral coefficients and linear prediction cepstral coefficients, and inputting the key data features into a pre-trained deep neural network to extract acoustic features;

[0042] Analyzing the speaker's emotional state using a speech emotion recognition algorithm according to the intonation pattern and the acoustic features, and generating data features of tone and intonation;

[0043] The rhythm pause analysis sub-step comprises:

[0044] Pre-emphasize the pre-processed audio signal and output the pre-emphasized audio signal;

[0045] Performing speech energy detection on the pre-emphasized audio signal, calculating the energy profile of the pre-emphasized audio signal, and identifying areas where the energy is lower than a preset threshold, marking them as silence or pause;

[0046] Performing spectral entropy analysis on the pre-emphasized audio signal to measure the entropy value of the spectrum to distinguish between speech segments and non-speech segments;

[0047] Extracting time domain features from the pre-emphasized audio signal, wherein the time domain features include a zero crossing rate ZCR;

[0048] Extracting frequency domain features from the pre-emphasized audio signal, wherein the frequency domain features include spectrum centroid and spectrum flatness, and are used to analyze spectrum characteristics of the audio signal;

[0049] Based on the extracted time domain features and frequency domain features, data features of rhythm pauses are generated for use in evaluating psychological states.

[0050] In some possible implementations, step S15 specifically includes step S153: processing the thermal imaging data in the multimodal data;

[0051] The step S153 comprises:

[0052] Preprocessing of thermal imaging data;

[0053] Using a thermal imaging processing algorithm to locate the thermal area of ​​the human body for the preprocessed thermal imaging data, and extracting body temperature data from the thermal area;

[0054] For the pre-processed thermal imaging data, based on the dynamic heat source, the thermal image changes of the mouth, nose and chest area are scanned to monitor the temperature changes caused by breathing. The temperature changes of the mouth, nose and chest as well as the temperature changes of the gas produced by breathing are analyzed through time series to calculate the respiratory frequency.

[0055] Heart rate monitoring by identifying microcirculatory changes in the facial area from preprocessed thermal imaging data;

[0056] The dynamic heat sources in the preprocessed thermal imaging data are analyzed, the human body is recognized using a motion detection algorithm, and morphological operations are performed to identify the human body contour.

[0057] In some possible implementations, step S15 specifically includes step S154: processing the near infrared data in the multimodal data;

[0058] The step S154 comprises:

[0059] Based on the Lambert-Beer law, the concentration change of hemoglobin is calculated according to the known incident light intensity and the captured outgoing light intensity;

[0060] By analyzing the changes in hemoglobin concentration, the activity data characteristics of different areas of the human frontal cerebral cortex are obtained.

[0061] In some possible implementations, step S15 specifically includes step S155: processing the high-speed photography data in the multimodal data;

[0062] The step S155 specifically includes:

[0063] Acquire black and white high-speed photography data;

[0064] Performing histogram equalization processing on the high-speed photography data and standardizing the image pixel values;

[0065] A multi-task convolutional neural network model is used to obtain facial features and extract multiple muscle features on the human face. These features are then input into a three-dimensional convolutional neural network to perform micro-expression recognition on time series data. The output is a multi-dimensional feature tensor to obtain micro-expression data.

[0066] The pupil-corneal reflection method is used to illuminate the eyes with infrared light and collect infrared light reflected from the cornea and retina. The pupil position and light spot detection algorithm are used to obtain the position of the pupil and the light spot on the cornea to determine the line of sight of the eye and obtain eye movement data.

[0067] In some possible implementations, step S15 specifically includes step S156: using transfer learning to input the processed multimodal data into the first personal psychological model to perform training iterative optimization to obtain a second personal psychological model.

[0068] In some possible implementations, step S15 specifically includes step S156, and step S156 includes the following steps:

[0069] performing transfer learning on a pre-trained base psychological model, the base psychological model serving as a first person psychological model;

[0070] Implementing a hierarchical freezing rule on preset layer data of the first personal psychological model, the preset layer data including a bottom layer, a top layer, and a preset middle layer of the first personal psychological model;

[0071] Using a phased unfreezing method, gradually unfreeze the preset middle layer and top layer of the first personal psychological model;

[0072] For the unfrozen preset middle layer and top layer, fine-tune the corresponding layer parameters using a first learning rate, and fine-tune the layer parameters of the unfrozen bottom layer of the first personal psychological model using a second learning rate, wherein the first learning rate is greater than the second learning rate, to obtain the first personal psychological model after the model parameters are fine-tuned;

[0073] Fusion of micro-expressions, facial expressions, pupil changes and body language features from video data; language content, intonation, tone and rhythm pause features from voice data; and body temperature, breathing rate, heart rate and human movement features from infrared data;

[0074] The data features after feature fusion are used for transfer learning, and the first person psychological model after fine-tuning the model parameters is iteratively optimized to obtain a second person psychological model.

[0075] In some possible implementations, the pre-trained large language model is a pre-trained large language model based on the Transformer architecture, which is used to construct a meta-model of a personal psychological model. The large language model contains 11 billion parameters distributed in 64 Transformer layers, each layer has multiple self-attention heads and feedforward networks, and is trained using a step-by-step distillation method. The training data set of the large language model contains text related to the field of mental health.

[0076] In a second aspect, an artificial intelligence psychological assessment system based on multimodal input is provided, the system comprising:

[0077] Personal psychological model training module, used to establish and train personal psychological models for psychological assessment;

[0078] A dialogue module, used for receiving a psychological assessment request initiated by a user, and conducting a dialogue with the user through the personal psychological model;

[0079] A multimodal data acquisition module, used to acquire multimodal data of the user during the conversation with the user;

[0080] The mental health assessment module is used to process and analyze the multimodal data of the user and conduct a mental health assessment on the user.

[0081] The above technical solution has the following beneficial technical effects:

[0082] This method can obtain and process multiple types of user data, such as text, voice, facial expressions, physiological signals, etc. This multimodal data fusion method can provide more comprehensive and accurate psychological state information, thereby improving the accuracy of psychological assessment. Multimodal input allows users to choose the most suitable interaction method for themselves, such as voice, text or video, thereby improving user participation and satisfaction.

[0083] This method builds and trains a personalized psychological model for each user, which can better understand the user's psychological state and needs and provide more targeted psychological assessment services. By communicating with the user, this method can obtain the user's psychological state and feedback in real time, adjust the psychological assessment strategy in time, and provide more flexible and personalized services.

[0084] Psychological assessment methods based on artificial intelligence technology can reduce the impact of human factors and provide more objective and standardized psychological assessment and treatment services. This helps ensure the fairness and consistency of services.

[0085] This method can serve as a powerful tool for mental health professionals to assist them in initial assessment, diagnosis or treatment recommendations. By automating the processing and analysis of multimodal data, it can reduce the workload of professionals and improve their work efficiency.

[0086] Under the premise of ensuring data security and privacy, this method can conduct psychological assessments without the user having to directly face a mental health expert, thereby protecting the user's privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0088] Figure 1 is a flow chart of an artificial intelligence psychological assessment method based on multimodal input according to an embodiment of the present invention;

[0089] Figure 2 is a flow chart of another artificial intelligence psychological assessment method based on multimodal input according to an embodiment of the present invention;

[0090] Figure 3 is a schematic diagram of multimodal data classification according to an embodiment of the present invention;

[0091] Figure 4 It is a functional block diagram of an artificial intelligence psychological assessment system based on multimodal input according to an embodiment of the present invention. DETAILED DESCRIPTION

[0092] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0093] Figure 1 is a flow chart of an artificial intelligence psychological assessment method based on multimodal input according to an embodiment of the present invention, such as Figure 1 As shown, the method specifically comprises the following steps:

[0094] S10: Build and train personal psychological models for psychological assessment.

[0095] S20: Receive a psychological assessment request initiated by a user and communicate with the user through a personal psychological model.

[0096] In some embodiments, the personal psychological model may provide a multi-round dialogue mode to implement multi-round replies based on the user's replies and start a dialogue. The dialogue may include open-ended questions designed to guide users to share their feelings, thoughts, and experiences. For example, "Have you encountered anything that has made you feel very stressed recently?" or "Can you describe the last time you felt very happy?" The dialogue may also include more specific questions designed to assess the user's psychological state, such as "Do you often feel helpless or hopeless?" or "Do you feel that your life has goals and direction?"

[0097] S30: Acquire multimodal data of the user during the conversation with the user.

[0098] Specifically, during a conversation, the generation of multimodal data can be achieved through different channels. For example, video data, audio signals, near-infrared data, thermal imaging data, and high-speed photography data can be collected in a video call. In real-time conversations, AI can prompt users to generate multimodal data through specific reactions or actions, for example, recording their expressions and voices when users talk about specific emotional experiences, or stimulating and recording these data through specific interactive games and tasks. Among them, video data is used to capture facial expressions and body language; audio signals are used to analyze the pitch, rhythm, and intensity of voice, which reflect a person's emotional state; near-infrared data is used to monitor changes in blood flow, which can be used as an indicator of emotional excitement or stress, thereby detecting emotional changes; thermal imaging data is used to identify changes in body temperature caused by stress or emotions, and changes in body temperature can reflect different emotional states, such as tension or relaxation; high-speed photography data is used to capture fast and subtle changes in facial expressions, which may be ignored in ordinary video capture.

[0099] S40: Process and analyze the user's multimodal data and conduct a mental health assessment on the user.

[0100] This step may specifically include the following steps:

[0101] S40-1: Data preprocessing, which includes audio processing, video processing, and other modality data processing.

[0102] Audio processing involves extracting audio features such as intonation, speech rate, pause duration, etc., which reflect emotions and stress levels. Video processing involves facial expression recognition,

[0103] Analyze body language, extract facial expression changes, etc. This includes identifying specific emotional expressions (such as happiness, sadness, anger). Other modality data processing refers to the preprocessing of near-infrared, thermal imaging and high-speed photography data. The corresponding preprocessing steps include image enhancement, feature extraction, etc., which are used to monitor physiological indicators related to emotional changes, such as heart rate changes, body temperature changes, etc.

[0104] S40-2: Perform feature extraction and fusion. The features extracted from each modality are fused to create a comprehensive feature set, which helps provide a comprehensive view of the mental state. Feature fusion can be performed at different levels, such as early fusion (fusion at the feature level) or late fusion (fusion at the decision level).

[0105] S40-3: Apply machine learning and deep learning models for emotion recognition, stress detection, and psychological state assessment.

[0106] Emotion recognition refers to the application of deep learning models (e.g., convolutional neural networks, recurrent neural networks) to identify emotional expressions in audio and video data. Stress detection refers to the use of machine learning algorithms (e.g., support vector machines, random forests, etc.) to identify physiological and behavioral signs of stress, combining features from multiple data sources. Mental state assessment refers to the analysis of comprehensive features through deep learning models to assess more complex mental health states, such as depression, anxiety, etc.

[0107] S40-4: Output analysis and evaluation results related to the user's mental health. The output of the model is interpreted as specific mental state indicators, such as the degree of mood swings, stress levels, and possible mental health problems. Based on the evaluation results, provide personalized suggestions and treatment plans, such as relaxation exercises, professional psychological counseling recommendations, etc.

[0108] Figure 2 is a flow chart of another artificial intelligence psychological assessment method based on multimodal input according to an embodiment of the present invention, such as Figure 2 As shown, the artificial intelligence psychological assessment method based on multimodal input includes the following steps: input of user personal basic information, which may include: name, gender, birthday, hobbies, occupation, facial biometric information and other personal information. The basic information is input into the personal psychological model creation module to create an initialized personal psychological model. When the user initiates psychological assessment and psychological therapy, the initialized personal psychological model generates a dialogue or inquiry discourse to have a dialogue with the user. In the process of having a dialogue with the user, the user's multimodal data is collected in real time, including any or all of video data, audio signals, near-infrared data, thermal imaging data, and high-speed photography data, and then these multimodal data are input into the initialized personal psychological model to iteratively strengthen training and optimization, and finally obtain a trained personal psychological model for psychological assessment.

[0109] In some embodiments, in some possible implementations, step S10 specifically includes:

[0110] S11: Optimize the pre-trained large language model and adapt it to the context of psychological assessment through step-by-step distillation;

[0111] S12: Initialize the pre-trained large language model based on the user's basic information to obtain the first person psychological model;

[0112] S13: Communicate with users through the first person psychological model;

[0113] S14: During the conversation, multimodal data of the user is collected, where the multimodal data includes video data, audio signals, near infrared data, thermal imaging data, and high-speed photography data;

[0114] S15: Processing multimodal data;

[0115] S16: Use transfer learning to input the processed multimodal data into the first person psychological model for iterative training optimization to obtain the second person psychological model.

[0116] In some embodiments, step S12 specifically includes the following steps:

[0117] S121: Collect basic information of the user, and encode the basic information of the user into a processing format of a large language model, where the encoding method includes at least one of one-hot encoding, embedding vector, and combined features;

[0118] S122: Load a large pre-trained language model based on the Transformer architecture from storage;

[0119] S123: Performing personalized parameter adjustment on the pre-trained large language model to obtain the first personal psychological model, which includes:

[0120] S123-1: Add one or more new neural network layers on top of the neural network of the pre-trained large language model to process personalized features;

[0121] Specifically, user basic information refers to the preliminary data provided by the user, such as age, gender, occupation, etc. After being encoded, this basic information can be used to generate or adjust the artificial intelligence model. Personalized features refer to features obtained by further analysis or processing based on user basic information. These features reflect the user's personality, habits or psychological state in more detail. Therefore, personalized features are deeper feature descriptions based on the user's basic information and obtained through model processing.

[0122] S123-2: The encoded basic information of the user is used as the input of the pre-trained large language model with a new neural network layer added, combined with the input of each Transformer layer of the pre-trained large language model, and outputs a set of adaptive weights;

[0123] Specifically, these encoded user basic information are used as the input of the pre-trained large language model. They are part of the input of each Transformer layer of the large language model and participate in the calculation together with the original input of the model.

[0124] Additional input refers to the fact that this encoded user basic information is not the input data originally designed for the pre-trained model. In the standard use of pre-trained models, the input is text data. Additional input means that additional user basic information is added as input on the basis of the original input to introduce personalized contextual information to help the model better understand and adapt to the characteristics and needs of specific users.

[0125] Adaptive weights are parameters used in machine learning and artificial intelligence that are adjusted based on the data or behavior of a specific user. These adaptive weights are designed to make the model better adapt to or reflect the specific characteristics or preferences of individual users. In the context of AI psychological assessment, adaptive weights can be based on the user's basic information (such as age, gender, occupation, etc.) and / or their behavioral data (such as browsing habits, interaction patterns, etc.), and after being specially encoded, are used to adjust pre-trained large language models to generate personalized psychological models.

[0126] S123-3: combining the basic weights and adaptive weights of the pre-trained large language model, calculating a new weight matrix by weighted summation, so as to personalize the pre-trained large language model and obtain a first personal psychological model; wherein the value range of the mixed parameter α of the mixed weighted summation is between 0 and 1;

[0127] Specifically, this step calculates a new weight matrix, which is obtained by combining the basic weights of the pre-trained model and the adaptive weights (weights based on basic user information). This is a hybrid weighting method. Here, the hybrid parameter α (value range is between 0 and 1) is used to adjust the importance or contribution of the basic weights and adaptive weights, and finally a new weight matrix is ​​obtained by weighted summation.

[0128] A new weight matrix is ​​determined by a mixed weighted summation method to personalize the pre-trained large language model to obtain the first personal psychological model, wherein the mixed parameter α of the mixed weighted summation ranges from 0 to 1.

[0129] Assume that there is a basic weight matrix Wb, an adaptive weight matrix Wa, and a mixing parameter α, where the value of α is between 0 and 1. The calculation formula of the new weight matrix Wn can be: Wn = α Wa + (1-α) Wb. Here, α is used to adjust the relative importance of the adaptive weight and the basic weight in the new weight matrix. If α is close to 1, the adaptive weight has a greater impact; if α is close to 0, the basic weight has a greater impact.

[0130] Assume that for a certain layer of weights, the basic weight matrix Wb is [0.2, 0.8], the adaptive weight matrix Wa is [0.5, 0.5], and the mixing parameter α is selected as 0.6. Then the new weight matrix Wn is calculated as follows: Wn = 0.6 · [0.5, 0.5] + 0.4 · [0.2, 0.8] = [0.38, 0.62]. This new weight matrix Wn will be used in the model to better reflect the characteristics of individual users.

[0131] S124: Based on the user basic information, independently store the first personal psychological model corresponding to the user.

[0132] Specifically, this step completes the initialization establishment and training of the first personal psychological model. The first personal psychological model is stored separately to ensure that only the individual who initializes the first personal psychological model can communicate with the first personal psychological model in the future.

[0133] Specifically, after the initialization and training of the personal psychological model is completed, its separate storage can be achieved through a database or file system, and each user model has a unique identifier (such as a user ID). In this way, when the user interacts with the system, the system will retrieve the corresponding personalized psychological model through the user's authentication information (such as login credentials). Ensuring that the individual and the corresponding model are in dialogue can be achieved in the following ways: the user needs to authenticate (such as log in) before interacting with the system; the system queries the corresponding personal psychological model based on the authenticated user ID; during the interaction process, all requests and responses are made through this specific model, ensuring the personalization and security of the interaction.

[0134] Figure 3 is a schematic diagram of multimodal data classification according to an embodiment of the present invention, such as Figure 3 As shown, in some embodiments, step S15 specifically includes step S151: processing the video data in the multimodal data;

[0135] Step S151 specifically includes: a video preprocessing step, a body movement data extraction step, a facial expression analysis step, and a pupil change detection step;

[0136] The video preprocessing steps specifically include:

[0137] Perform frame extraction on the video data, and decompose the video data into separate video frames;

[0138] The resolution of the video frame is adjusted to balance processing efficiency and analysis quality; specifically, the resolution of the video frame is adjusted to find a balance between processing efficiency and analysis quality. The resolution of the original video may be very high, such as 4K (3840x2160) or Full HD (1920x1080), which consumes a lot of computing resources when processing, especially when the video data is used for deep learning model training or real-time analysis. In order to maintain reasonable processing time and resource usage, the resolution will be adjusted to between 720x576 (standard definition) and 1280x720 (high definition). The specific value depends on the requirements of the target task and the limitations of computing resources. The step of adjusting the resolution may specifically include: identifying the accuracy requirements of the video analysis task. For example, facial recognition may require a higher resolution, while motion detection may have lower requirements for resolution. Determine the most appropriate resolution based on available computing resources (such as the computing power of a GPU). Use an image processing library (such as OpenCV) to adjust the resolution of the video frame.

[0139] Normalize the video data to standardize the brightness and contrast of the video frames. Specifically, the specific implementation of normalization may include the following steps: Calculate the average brightness and contrast: Calculate the brightness and contrast of all frames in the video to find an average value or target value; Adjust the brightness and contrast of each video frame to be close to the average value or target value. This can be achieved through various image processing techniques, such as histogram equalization or other image enhancement techniques. The purpose of video data normalization is to reduce the changes in visual features of video data, make the model easier to learn, and improve the generalization ability of the model. In an embodiment of the present invention, it mainly includes standardizing the brightness and contrast of the video frames to reduce the impact caused by changes in illumination. The normalized video frames will have more consistent brightness and contrast levels. All video frames will be adjusted to a similar brightness and contrast range without being affected by illumination conditions.

[0140] The limb motion data extraction step specifically includes: using a human posture estimation algorithm to detect and track key points of the human body in the processed video frames, and identifying specific limb motions and behavior patterns by analyzing the movement of key points over time to generate limb motion data features;

[0141] The facial expression analysis step specifically includes: using a facial detection algorithm to identify the facial area in the preprocessed video frame, and using a facial feature point detection algorithm to track the key feature points of the face to obtain the facial expression information of the current user;

[0142] Among them, the pupil change detection step specifically includes: using the eye area positioning algorithm to determine the points related to the eyes among the facial feature points in the preprocessed video frame, and using the pupil positioning algorithm to determine the exact position of the pupil, and then performing pupil size change analysis to generate pupil change data features.

[0143] The video preprocessing step is performed first, which establishes a high-quality data foundation for the subsequent analysis step. This is a preparatory stage to ensure that the video data is presented in a format that is easier to analyze. The body movement data extraction step, the facial expression analysis step, and the pupil change detection step are performed after the video preprocessing step. In order to improve data processing efficiency, in an embodiment of the present invention, these three steps can be performed in parallel. In other embodiments, serial processing can also be adopted.

[0144] In some embodiments, step S15 specifically includes step S152: processing the audio signal in the multimodal data; step S152 specifically includes: an audio preprocessing sub-step, a speech recognition sub-step, a tone and intonation analysis sub-step, and a rhythm pause analysis sub-step;

[0145] In some embodiments, the audio preprocessing sub-step includes: performing noise reduction processing on the audio signal, the noise reduction processing includes: removing background noise and noise, and performing echo cancellation and gain control to standardize the volume; segmenting the audio signal after noise reduction processing to obtain audio frames that have been processed by noise reduction and segmentation; extracting spectral features from the audio frames that have been processed by noise reduction and segmentation, and normalizing the extracted spectral features. Through normalization processing, the consistency of the data is ensured, which provides preparation for subsequent speech recognition, tone and intonation analysis, and rhythm pause analysis.

[0146] In some embodiments, the speech recognition sub-step includes: taking the preprocessed audio signal as input, using a sequence-to-sequence model to perform speech recognition, and outputting a preliminary text sequence corresponding to the preprocessed audio signal; wherein the sequence-to-sequence model includes an encoder and a decoder with an attention mechanism, and introduces a reinforcement learning strategy, using the minimum word error rate as a reward function for model training; taking the preprocessed audio signal and the preliminary text sequence as input, performing joint training and joint decoding in combination with a converter model, and outputting an optimized text sequence.

[0147] Among them, a sequence-to-sequence model is used for speech recognition. The input of this step is the audio signal processed by the audio preprocessing sub-step. Preprocessing includes noise reduction, segmentation into short time frames, feature extraction, etc., which aims to improve the quality of audio signals for more accurate speech recognition. The sequence-to-sequence model includes an encoder and a decoder with an attention mechanism. The encoder first converts the preprocessed audio signal into a series of internal representations. The decoder uses these representations and focuses on the input part that is most relevant to the current prediction through the attention mechanism to gradually generate a text sequence.

[0148] Among them, the reinforcement learning strategy is introduced, and the minimum word error rate is used as the reward function for model training, aiming to optimize model performance and reduce recognition errors. The output of this step is the text sequence converted from the audio signal, that is, the language content in the audio is recognized and transcribed into text.

[0149] Among them, the input of the joint training and joint decoding steps combined with the transformer model is the audio signal processed by the audio preprocessing sub-step and the output of the sequence-to-sequence model (i.e., the initially recognized text sequence). The transformer model is a deep learning model that relies on the self-attention mechanism to process sequence data. The transformer model is jointly trained with the sequence-to-sequence model, and the two models share data, learning process, and goals. Joint decoding means that in actual applications, the two models work together to improve the accuracy and efficiency of speech recognition by complementing each other. The output of this step is a further optimized text sequence. The introduction of the transformer model and the joint training and decoding with the sequence-to-sequence model can improve the accuracy of speech recognition, reduce the word error rate, and ultimately output more accurate and fluent text content.

[0150] In some embodiments, the tone and intonation analysis sub-step includes:

[0151] Use the pitch detection algorithm to extract the intonation pattern in the preprocessed audio signal, and analyze the intonation changes in combination with the results of speech recognition to identify different types of sentences;

[0152] Extracting key data features from the preprocessed audio signal using Mel frequency cepstral coefficients and linear prediction cepstral coefficients, and inputting the key data features into a pre-trained deep neural network to extract acoustic features;

[0153] Analyzing the speaker's emotional state using a speech emotion recognition algorithm according to the intonation pattern and the acoustic features, and generating data features of tone and intonation;

[0154] In some embodiments, the rhythm pause analysis sub-step includes:

[0155] Pre-emphasize the pre-processed audio signal and output the pre-emphasized audio signal, in which the high-frequency component is enhanced;

[0156] Performing speech energy detection on the pre-emphasized audio signal, calculating the energy profile of the pre-emphasized audio signal, and identifying areas where the energy is lower than a preset threshold, marking them as silence or pause; the output of this step is an audio signal with silence or pause areas marked;

[0157] Performing spectral entropy analysis on the pre-emphasized audio signal to measure the entropy value of the spectrum to distinguish between speech segments and non-speech segments; this step outputs the audio signal that distinguishes between speech segments and non-speech segments;

[0158] Extracting time domain features from the pre-emphasized audio signal, wherein the time domain features include a zero crossing rate ZCR, which is a signal feature used to reflect and assist in detecting pauses;

[0159] Extracting frequency domain features from the pre-emphasized audio signal, wherein the frequency domain features include spectrum centroid and spectrum flatness, and are used to analyze spectrum characteristics of the audio signal;

[0160] Based on the extracted time domain features and frequency domain features, data features of rhythm pauses are generated for use in evaluating psychological states.

[0161] In some embodiments, step S15 specifically includes step S153: processing the thermal imaging data in the multimodal data;

[0162] The step S153 comprises:

[0163] Infrared video preprocessing: First, the thermal imaging data is frame extracted and the continuous video stream is decomposed into single frames for frame-by-frame analysis. Then, the frame resolution is adjusted to balance the processing efficiency and analysis quality, and the frames are normalized to reduce the impact of ambient temperature changes. In addition, background subtraction technology is used to remove the impact of static heat sources, thereby highlighting the dynamic heat source, namely the human body, and preparing data for subsequent body temperature data extraction, respiratory rate and heart rate detection, and human body detection steps.

[0164] Body temperature data extraction: Based on infrared video preprocessing, a thermal imaging processing algorithm is used to locate the hot areas of the human body, especially the facial area. Then, accurate temperature readings are obtained by calibrating with a pre-stored standard temperature source, and body temperature data is extracted from the hot areas.

[0165] Respiratory rate detection: Based on the prominent dynamic heat source, the pre-processed thermal imaging data is scanned for thermal image changes in the mouth, nose, and chest area, and the temperature changes caused by breathing are monitored. The respiratory rate is calculated through time series analysis of the temperature changes in the mouth, nose, and chest as well as the temperature changes of the gases produced by breathing.

[0166] Heart rate detection: Heart rate monitoring is performed by identifying microcirculation changes in the facial area in infrared video. Specifically, temperature fluctuations in the area under the nose or other vascular-rich areas are detected, and the frequency of temperature fluctuations is analyzed using signal processing algorithms (such as fast Fourier transform) to estimate the heart rate.

[0167] Human body detection: Analyze dynamic heat sources in the video, such as moving human bodies, use motion detection algorithms to identify human bodies, and perform morphological operations (such as dilation and erosion) to improve the recognition of human body contours to optimize the accuracy and reliability of subsequent steps.

[0168] In some embodiments, step S15 specifically includes step S154: processing near infrared data in the multimodal data;

[0169] Step S154 includes:

[0170] Based on the Lambert-Beer law, the concentration change of hemoglobin is calculated according to the known incident light intensity and the captured outgoing light intensity;

[0171] By analyzing the changes in hemoglobin concentration, the activity data characteristics of different areas of the human frontal cerebral cortex are obtained.

[0172] This activity level data can then be input into the first person psychological model in the psychological assessment system to evaluate changes in blood oxygen levels in areas of neural activity in the brain during cognitive activities.

[0173] Specifically, the known incident light intensity refers to the initial intensity or amount of near-infrared light incident on the cerebral cortex. This intensity is preset and known before the experiment or measurement begins. In some embodiments, the incident light intensity is preferably 760nm and 850nm, which are sensitive to deoxyhemoglobin and oxyhemoglobin, respectively. The incident light intensity can be accurately set by controlling the power of the light source.

[0174] The captured outgoing light intensity refers to the intensity of the near-infrared light emitted from the surface of the scalp after being scattered and absorbed by the cerebral cortex. This intensity is captured by a dedicated near-infrared camera, using narrow-band 760nm and narrow-band 850nm cameras to capture the corresponding wavelengths of outgoing light respectively. Changes in the outgoing light intensity reflect the light absorption and scattering inside the brain, which is directly related to changes in hemoglobin concentration. Changes in hemoglobin concentration are caused by changes in blood flow and blood oxygen levels caused by brain activity.

[0175] The Lambert-Beer law allows the concentration changes of deoxyhemoglobin and oxyhemoglobin to be calculated based on the known incident light intensity and the captured outgoing light intensity. The Lambert-Beer law describes that the intensity of light passing through a medium is attenuated in direct proportion to the concentration of light-absorbing substances in the medium. The calculation of the changes in hemoglobin concentration is used to estimate the activity of certain areas of the brain, especially different areas of the prefrontal cortex. This information can be used to assess the changes in blood oxygen levels in areas of neural activity in the brain during cognitive activities, providing important clues about the individual's psychological state and health.

[0176] Inputting data on brain activity into the personal psychological model of the psychological assessment system can help evaluate brain neural activity during cognitive activities, such as concentration, emotional response, etc. Analyzing this data allows the system to identify possible abnormalities in psychological states, such as anxiety, depression, or cognitive dysfunction, and provide personalized treatment suggestions or intervention measures accordingly.

[0177] In some embodiments, step S15 specifically includes step S155: processing the high-speed photography data in the multimodal data;

[0178] Step S155 specifically includes:

[0179] Acquire black and white high-speed photography data;

[0180] The black and white high-speed photography data is subjected to histogram equalization to obtain a black and white image, and the pixel value of the black and white image is scaled to a range of 0 to 1 for standardization; specifically, the image pixel value refers to the pixel value of the black and white image obtained from the high-speed photography data. These images are mainly used to extract eye movement data and micro-expression data.

[0181] A multi-task convolutional neural network model is used to obtain facial features from high-speed photography data images after histogram equalization and standardization, and multiple muscle features of the human face are extracted. The facial features extracted from the multi-task convolutional neural network model are input into a three-dimensional convolutional neural network to perform micro-expression recognition on time series data, and the output is a multi-dimensional feature tensor to obtain micro-expression data; these feature tensors represent the characteristic information of micro-expressions.

[0182] The pupil-corneal reflection method is used to illuminate the eyes with infrared rays and collect infrared rays reflected from the cornea and retina. The pupil location and light spot detection algorithm (such as the Starburst algorithm) are used to obtain the pupil position and the position of the light spot on the cornea from the infrared reflection image to determine the line of sight of the eye movement and obtain eye movement data. The pupil position and the position of the light spot on the cornea are obtained by using the pupil-corneal reflection method. This technology involves illuminating the eyes with infrared rays and collecting infrared rays reflected from the cornea and retina. The Starburst algorithm is used to obtain the pupil position and the position of the light spot on the cornea from these infrared reflection images.

[0183] Furthermore, the feature tensor of the micro-expression and the gaze direction data of the eye movement are input into the first person psychological model in the psychological assessment system for prediction and training to assess the individual's psychological state. In this way, the system can use the subtle facial expressions and eye movement information extracted from high-speed photography data to assess and understand the individual's psychological state, and then provide accurate mental health assessment and corresponding treatment suggestions.

[0184] In some embodiments, step S15 specifically includes step S156: using transfer learning to input the processed multimodal data into the first personal psychological model to perform training iterative optimization, thereby obtaining a second personal psychological model;

[0185] Step S156 includes the following steps:

[0186] Transfer learning initialization: Implement transfer learning on a pre-trained basic psychological model. This basic psychological model has been pre-trained on a wide range of data and has general psychological recognition capabilities.

[0187] Layer Freezing Rules: Layer freezing rules are implemented on the preset layer data of the first person mental model, including the bottom layer, top layer, and preset middle layers of the first person mental model base mental model. Layer Freezing Rules is a technique in transfer learning that is used to control which layers of a neural network are updated during training. In the context of transfer learning, a model pre-trained on a large dataset is used as a starting point, and then fine-tuned to fit a specific task or dataset. The bottom layers are responsible for extracting general features, and in transfer learning, the weights of these layers are frozen, i.e., not updated during training, because they are already able to capture useful features. The top layers are closer to the output and are responsible for extracting features that are more relevant to a specific task. In transfer learning, the weights of these layers are unfrozen, allowing them to be updated on the new dataset to better fit the new task. The middle layers may contain both general features and task-specific features. Depending on the complexity of the task and the size of the new dataset, the weights of these layers can be selectively frozen or unfrozen. Phased unfreezing: A phased unfreezing approach is adopted, that is, more preset middle layers and top layers (high-level layers close to the output of the model) of the first person's psychological model are gradually unfrozen for fine-tuning so that the model can better adapt to the individual's data characteristics. These fine-tuned layers focus on capturing more detailed features related to the individual's psychological state.

[0188] Learning rate adjustment: For the unfrozen preset middle layer and top layer, the first learning rate is used to fine-tune the corresponding layer parameters, and the second learning rate is used to fine-tune the layer parameters of the bottom layer of the unfrozen (still frozen) first person psychological model, wherein the first learning rate is greater than the second learning rate, and the first person psychological model after the model parameters are fine-tuned is obtained. The learning rate is adjusted through learning rate decay or cyclic learning rate strategy to optimize the model learning process. The learning rate is a hyperparameter that controls the update amplitude of the model weight. Since the bottom layer contains common features, a smaller learning rate can be used to retain the knowledge in the pre-trained model. Since the top and middle layers need to adapt to new tasks, a larger learning rate can be used to speed up the learning of new features. An excessively large learning rate may cause the model to be unstable or even divergent during the training process, thereby affecting the model performance. An excessively small learning rate may cause the model to learn too slowly, requiring more training time, or fall into a local optimum and fail to achieve optimal performance.

[0189] Multimodal data feature fusion: Fusion of micro-expressions, facial expressions, pupil changes, and body language features from videos; language content, intonation, tone, and rhythm pause features from speech; and body temperature, respiratory rate, heart rate, and human motion features from infrared data. These data features come from the preprocessing and analysis of multimodal input data, including video, audio, and infrared thermal imaging data. Feature fusion is the combination of data features from different sources to improve the performance of the model. Feature fusion methods include: Early Fusion, which directly splices data from different sources together at the feature level and then inputs them into the model for training; Late Fusion, which trains the model of each data source separately and then combines the outputs of different models at the decision level, such as by voting, averaging, or weighted averaging; Intermediate Fusion, which combines features from different sources at the intermediate layer of the model.

[0190] Transfer learning and model optimization: Use the data features after feature fusion for transfer learning, iteratively optimize the first person psychological model after fine-tuning the model parameters, and obtain an advanced psychological model optimized for a specific user, that is, the second person psychological model. This process ensures that the model can accurately identify and evaluate the psychological state of a specific user.

[0191] The above technical solution is described in detail below:

[0192] 1. Establishing a personal psychological model and building a psychological profile

[0193] The present invention discloses an artificial intelligence psychological assessment method and system based on multimodal input, the method comprising the following steps: optimizing a pre-trained large language model by a stepwise distillation method to adapt to the context of psychological assessment; initializing a personal psychological model according to basic information of a user to obtain a first personal psychological model; the first personal psychological model communicates with the user by generating a dialogue to complete the psychological assessment task, and simultaneously obtains the multimodal data of the user during the dialogue, the multimodal data including video data, audio signals, near infrared data, thermal imaging data and high-speed photography data during the dialogue, and inputs the processed multimodal data into the first personal psychological model by a transfer learning method to train and iterate and optimize it, completes the fine-tuning of the first personal psychological model, and obtains a second personal psychological model; each time the personal psychological model communicates with the user, the process from the first personal psychological model to the second personal psychological model is completed, and the personal psychological model is continuously iteratively optimized. By combining language communication and audio and video, near infrared, thermal imaging, and high-speed photography data, the personal psychological model can provide a comprehensive psychological health assessment.

[0194] 2. Model Iteration Optimization

[0195] 2.1. Pre-training large language models

[0196] The embodiment of the present invention uses a pre-trained large language model based on the Transformer architecture. This model contains 11 billion parameters, which are distributed in 64 layers of Transformer, and each layer has multiple self-attention heads and feedforward networks. This language model is the basis of all personal psychological models, that is, the meta-model of all psychological models. In the process of training the model, a step-by-step distillation method is adopted, that is, a larger model is used to generate a labeled interpretable problem, so that the small model not only needs to predict the output label but also needs to provide a reasonable explanation for the problem. The Transformer architecture was introduced by Vaswani et al. in 2017, and it is mainly based on the self-attention mechanism. This architecture does not rely on traditional recurrent neural networks (RNN) or convolutional neural networks (CNN), but directly captures the relationship between each element in the input sequence through the attention mechanism. The meta-model of the pre-trained large language model based on the Transformer architecture of the embodiment of the present invention has 11 billion parameters. The number of parameters is a key indicator for measuring the complexity and learning ability of the model. The more parameters there are, the higher the data complexity that the model can capture, but more data is also required for effective training. At the same time, each of the 64 Transformer layers contains a self-attention mechanism and a feedforward network, which can better fit the personal psychological model and effectively prevent overfitting. In the self-attention mechanism, the attention head can process data in parallel to capture the relationship between different positions in the sequence. Multiple attention heads can learn the characteristics of input data from different angles. At the same time, each Transformer layer also contains a feedforward network (FFN). The meta-model is trained with more than 50GB of data, including text datasets from the Internet (Wikipedia, Baidu Encyclopedia, etc.), as well as a large number of e-book datasets, as well as related academic papers and works, especially in the field of mental health.

[0197] The psychological language model distillation method provided in the embodiment of the present invention is applicable to all common large models, including Chatgpt, LLaMA, chatGLM, etc. Common large models should strictly follow the standard input and output process, and the generated explainable questions will be stored in a dictionary data structure, that is, {("question":"TEXT"), ("answers":"TEXT"), ("explain","TEXT")}.

[0198] 2.2 Personal psychological model initialization

[0199] After the user's basic information is collected, it is encoded into a format that can be processed by the model. Here, one-hot encoding (each attribute is represented by a one-hot vector, one position is set to 1 to represent the specific category of the user, and the rest of the positions are 0), embedded vectors (using a pre-trained embedding model for each attribute to convert text information into a dense vector representation) and combined features (combining the encoding of all basic information into a feature vector for initializing the personal psychological model) are used to generate personal basic information features. After the encoded information is completed, use the following steps to initialize:

[0200] The first step is to load the pre-trained Transformer model with 11 billion parameters from the storage; the second step is to perform personalized parameter adjustment, based on the pre-trained Transformer model, to adjust the parameters to adapt to personal information. Adapting to personal information means adjusting the parameters of the model so that it can better understand and predict the psychological state and behavior of a specific user. This adjustment is based on the basic information provided by the user (such as age, gender, occupation, etc.), as well as other information (such as historical behavior data, preference settings, etc.). This information is encoded in the form of one-hot encoding, embedding vectors, and combined features, and then used to personalize the parameters of the pre-trained Transformer model.

[0201] After adjusting the parameters, the initialization and training of the personal psychological model is completed. Such a personal psychological model will be stored separately, which is independent of the storage of the meta-model. At the same time, only the individual who initialized the model can communicate with the model in the future, which can effectively avoid data leakage.

[0202] 2.3. Continuous iterative optimization training

[0203] 2.3.1 Video Data Processing

[0204] Video preprocessing:

[0205] First, the video is frame extracted to decompose the video into individual frames for frame-by-frame analysis. Then the resolution of the video frame is adjusted to maintain a balance between processing efficiency and analysis quality. Finally, the video data is normalized to standardize the brightness and contrast of the video frame to reduce the impact of lighting changes.

[0206] Body movement data extraction:

[0207] First, human posture estimation is performed. A posture estimation algorithm (OpenPose is used here) is used to detect and track the key points of the human body in the video. Then, the movement of key points over time is analyzed to identify specific limb movements and behavior patterns to complete action recognition and extract limb movement data, generating limb movement data features.

[0208] Facial expression analysis:

[0209] First, a facial detection algorithm is used. Here, a multi-task cascaded convolutional neural network (MTCNN) is used to identify the facial area in the video frame. After the facial area is identified, facial feature point tracking is performed, and the facial feature point detection algorithm (Facial Action Coding System, FACS) is used to track the key feature points of the face. The key feature points are obtained and judged using the MTCNN model to obtain the current user's facial expression information (happy, sad, angry, etc.).

[0210] Pupil change detection:

[0211] First, the facial detection algorithm used in facial expression recognition can be used to locate the eye area, and the points related to the eyes in the facial feature points can be determined to locate the eye area. The pupil is located in the eye area, and the pupil location is determined using the pupil location and spot detection algorithm (for example, the Starburst algorithm). The size of the pupil is measured for each frame of the pupil position. By tracking the change of the pupil size over time, the user's psychological state such as disguise, excitement, interest, etc. can be determined, and the pupil change data feature can be generated.

[0212] 2.3.2 Audio Signal Processing

[0213] Audio Preprocessing:

[0214] First, the audio signal is segmented to divide the continuous audio signal into short time frames, for example, into segments of 20-40 milliseconds in length, for subsequent frame-by-frame analysis. Noise reduction is performed on each short-time frame audio signal to remove background noise and noise to improve the clarity of the audio signal. In some embodiments, the audio signal can also be echo-cancelled and gain-controlled to standardize the audio signal and reduce the impact of the recording environment. Finally, feature standardization is performed to normalize the spectral features to ensure that the model is not affected by differences in recording equipment.

[0215] Voice Recognition:

[0216] For the aforementioned pre-processed audio signal, the embodiment of the present invention can use the Seq2Seq model for speech recognition. The Seq2Seq model consists of two parts: an encoder and a decoder with an attention mechanism. When decoding each word, the attention mechanism dynamically calculates the weight of each input hidden state and obtains the current attention vector through weighted linear combination. In the speech recognition task here, the Seq2Seq model is similar to the Seq2Seq model in machine translation, and different models can be used as encoders and decoders, such as RNN, Transformer models, etc. At the same time, the embodiment of the present invention uses an enhanced learning strategy, takes the minimum word error rate as a reward function for model training, updates model parameters, and improves model robustness. Since the input and output of speech are monotonic, there is no reordering problem in machine translation, so the use of a monotonic attention strategy can improve the performance of speech recognition in some experiments. At the same time, the embodiment of the present invention is jointly trained and jointly decoded with the converter model Transducer to further increase the effect of speech recognition.

[0217] Tone and Intonation Analysis:

[0218] A pitch detection algorithm, such as the YIN algorithm, is used for the preprocessed audio signal to extract the intonation (pitch) pattern in the audio signal, and the changes in intonation are analyzed in combination with the results of speech recognition to identify questions, declarative sentences, exclamatory sentences, etc. After extracting the key data features of speech using Mel-frequency cepstral coefficients (MFCC) and Linear Predictive Cepstral Coefficients (LPCC), they are input into the pre-trained deep neural network to extract the acoustic features of the audio signal, such as rhythm, intensity, timbre, etc., which are related to tone, and a speech emotion recognition algorithm is used to analyze the speaker's emotional state to generate data features of tone and intonation.

[0219] Rhythm pause analysis:

[0220] Pre-emphasis is performed on the pre-processed audio signal, and the high-frequency components are enhanced through filters, because high frequencies usually contain important voice information. Then, voice energy detection is performed on the voice information, and the energy contour of the voice information is calculated. Areas with energy below a certain threshold are determined to be silence or pauses. Spectral entropy analysis is then performed on the voice information to measure the entropy of the spectrum of the voice information. A higher entropy value may indicate a non-speech segment, and a lower entropy value may indicate a speech segment. The time domain features of the processed voice information are extracted, mainly the zero-crossing rate (ZCR), which reflects the change in instantaneous energy in the audio signal and helps detect pauses. Frequency domain features are then extracted, mainly the spectrum centroid and spectrum flatness, which are used to analyze the spectrum characteristics of the audio signal and generate data features of rhythmic pauses.

[0221] The principle of pre-emphasis is to enhance the high-frequency part of the audio signal through a high-pass filter. The high-frequency components contain important speech information, such as consonants with high clarity and recognition. Pre-emphasis helps to strengthen these components, making subsequent analysis more effective.

[0222] 2.2.3 Thermal imaging data processing

[0223] Infrared video preprocessing corresponding to thermal imaging data:

[0224] First, the thermal imaging data captured by the infrared camera is frame extracted and decomposed into single frames for frame-by-frame analysis. The frame resolution is adjusted to balance processing efficiency and analysis quality, and the frames are normalized to reduce the impact of ambient temperature changes. At the same time, since infrared video is sensitive to temperature, it is necessary to use background subtraction technology to remove the impact of static heat sources and highlight the dynamic heat source, namely the human body.

[0225] Body temperature data extraction:

[0226] First, a thermal imaging processing algorithm is used to locate the hot areas of the human body, especially the facial area, which is usually an ideal area for body temperature measurement. Then, accurate temperature readings are obtained by calibrating with standard temperature sources pre-stored in the database, and body temperature data is extracted from the hot areas.

[0227] Respiratory rate detection:

[0228] For the preprocessed thermal imaging data, based on its prominent dynamic heat source, the thermal image changes of the mouth, nose and chest areas are scanned to monitor the temperature changes caused by breathing. At the same time, the breathing frequency is calculated through time series analysis of the mouth, nose and chest temperatures and the temperature changes of the gases produced by breathing.

[0229] Heart rate detection:

[0230] Heart rate monitoring is performed through facial microcirculation recognition. Specifically, microcirculation changes in the facial area are identified in infrared videos. This can be achieved by detecting temperature fluctuations in the area under the nose or other vascular-rich areas of the face. Signal processing algorithms are then used (fast Fourier transform can be used here) to analyze the frequency of temperature fluctuations to estimate the heart rate.

[0231] Human body detection:

[0232] Analyze dynamic heat sources in the video, such as moving human bodies, identify the human body using a motion detection algorithm, and perform morphological operations such as dilation and erosion on the detected heat sources to improve the recognition of human contours.

[0233] 2.2.4 Near-infrared data processing

[0234] Near infrared data description and processing:

[0235] The brain provides the oxygen required for neuronal activity through blood metabolism, and the consumption of oxygen stimulates the dilation of local blood vessels in the brain, which increases capillary blood flow, leading to an increase in local cerebral blood flow and cerebral blood volume, which is manifested as a rapid increase in the blood oxygen level in the brain. This is neurovascular coupling. Under the action of this mechanism, during cognitive neural activity, the oxygen carried by the increased cerebral blood flow in the brain's neural activity area will greatly exceed the oxygen required for brain activity, and oxygen is transported through hemoglobin in the blood. Therefore, during cognitive activity, the concentration of oxygenated hemoglobin (oxyHb) in the blood will increase and the concentration of deoxyhemoglobin (deoxyHb) will decrease in the brain's active area. The blood oxygen level will affect its optical properties. Physiological tissues have two reactions to light: one is absorption and the other is scattering. The main components of blood (water, oxygenated hemoglobin and deoxyhemoglobin) have very little absorption of near-infrared light of 600-900nm and have good scattering properties. Therefore, near-infrared light of 600-900nm is also called the spectral window. In this spectral window, near-infrared light around 760nm and 850nm has different sensitivities to oxyhemoglobin and deoxyhemoglobin. 760nm near-infrared light is sensitive to deoxyhemoglobin, while 850nm near-infrared light is more sensitive to oxyhemoglobin. Because the amount of scattering in different cortical layers of the head does not change due to neural activity, the attenuation caused by scattering in cortical tissue is considered constant. Therefore, the change in attenuation measured during cognitive activity is considered to be due to changes in absorption, which in turn is caused by changes in oxyhemoglobin and deoxyhemoglobin in brain activity tissue. Therefore, by irradiating 760nm and 850nm near-infrared light into the cerebral cortex, the incident light is captured using narrow-band 760nm and narrow-band 850nm near-infrared cameras. At this time, the incident light intensity and the outgoing light intensity are known, and the hemoglobin concentration can be estimated using the Lambert-Beer law.

[0236] Therefore, the original data obtained by the embodiment of the present invention is the fill light data captured by the narrow-band 760nm and narrow-band 850nm near-infrared cameras. The activity levels of different areas of the human frontal cerebral cortex are obtained by calculation, and this data is directly input into the model.

[0237] 2.2.5 High-speed photography data processing

[0238] Data preprocessing:

[0239] In an embodiment of the present invention, 640x480 black and white 1200 frames of high-speed photography data can be used to extract eye movement data and micro-expression data. Preferably, the frame rate of the data should be at least 1200 frames or more. The image contrast is improved by using histogram equalization, which can make the information of black and white pictures clearer. The image pixel values ​​are scaled to 0-1 for standardization. After preprocessing, such data can be input into the model for processing.

[0240] Micro-expression recognition:

[0241] The human face has 43 muscles, and different combinations of them will have different expressions. Therefore, the first thing to do is to extract the features of the 43 muscles. Here, the MTCNN model is used to obtain facial features and extract the features of the 43 muscles. The features are input into the three-dimensional convolutional neural network to perform micro-expression recognition and extraction on the time series data. The micro-expression obtained here is a combination of the movements of the 43 muscles, such as the twitching of eye muscle 1. In this way, each micro-expression is output as a 43-dimensional feature tensor, which can be input into the meta-model for prediction and training.

[0242] Eye movement recognition:

[0243] Eye movement recognition mainly uses the pupil corneal reflection (PCCR) technology, which is also called Pupil Center Corneal Reflection (PCCR) by some scholars. The basic principle of this technology is: use infrared light to illuminate the eyes; use a camera to collect infrared light reflected from the cornea and retina; due to the physiological structure and physical properties of the eyeball, under the premise that the relative position of the light source and the head remains unchanged, the light spot formed by the corneal reflection will not move, and the direction of the light reflected on the retina indicates the direction of the pupil (the light source enters from the pupil, and the retina reflects the light from the pupil); finally, the direction of the eye movement can be calculated according to the angle between the cornea and the pupil reflection light. In the embodiment of the present invention, a near-infrared fill light is added to the near-infrared, so for the data pre-processed by high-speed photography, the embodiment of the present invention only needs to perform feature extraction, that is, use the Starburst algorithm to obtain the pupil position and the position of the light spot on the cornea. These two positions can be represented by (x1, y1) and (x2, y2), and the line of sight direction is determined by the position of the pupil center relative to the corneal reflection. In this way, the embodiment of the present invention can obtain the gaze direction of each instantaneous eye movement, and this gaze direction data can be input into the model for prediction and training.

[0244] Specifically, the process of determining the sight direction using the pupil center corneal reflection technology in the embodiment of the present invention can be illustrated by the following example: First, an infrared light source is used to illuminate the eye, and a camera is used to collect the infrared light reflected by the eye. Here, the infrared light is reflected from two different points (cornea and retina) to form two bright spots. Through image processing technology, the two reflection points can be identified. One point is the corneal reflection point (fixed), and the other point is the reflection point of the pupil center (moves with the movement of the eye's sight). The position of the corneal reflection point is represented as: (x1, y1). The position of the reflection point of the pupil center is represented as: (x2, y2). Calculate the sight direction, which is determined by the position of the pupil center relative to the corneal reflection. The sight direction can be determined by calculating the vector between the two points, that is, the vector (x2-x1, y2-y1). This vector indicates the direction of the eye's gaze point relative to the eye position. After determining the sight direction, this information can be used in a variety of applications, such as eye tracking, user interface interaction, visual attention analysis, etc. By inputting these sight direction data into the machine learning model, further analysis and prediction can be performed.

[0245] 2.2.6 Feature Fusion

[0246] First, data alignment is performed, and timestamps are used to ensure data synchronization, followed by feature fusion. Three different models are used here for feature fusion. The first method is feature-level fusion, in which the various data features obtained above are fused using feature transformation. Here, principal component analysis (PCA) and independent component analysis (ICA) are used to reduce feature dimensions, while trying to retain information across multiple modalities and completing feature fusion at the same time. The second method is decision-level fusion, which means that at the model output level, each model makes predictions independently and then merges these decisions. The embodiment of the present invention uses a decision template, that is, creates a template, combines the outputs of different models, and assigns a score to each possible decision. The score here is gradually determined in continuous training and optimization, and is a hyperparameter (referring to an external configuration variable that is not learned in the machine learning model but manually configured). Finally, the third fusion performs model-level fusion. The multimodal deep learning model designed in the embodiment of the present invention can obtain the above data features through different input layers, and use the fusion layer to perform model-level fusion of data features.

[0247] In the embodiment of the present invention, feature fusion aims to integrate data features from different sources to improve the accuracy and efficiency of the model. This embodiment performs feature fusion on multiple modal data, covering visual, auditory and physiological signals, including: micro-expressions and facial expressions of video information; pupil changes and body language; language, intonation, tone and rhythm pauses of voice information; body temperature, respiratory rate, heart rate and human body detection of infrared information.

[0248] The advantage of feature fusion in this embodiment is that different data modalities can provide complementary information, and the fused features can more comprehensively reflect the individual's psychological state. Fusion of multimodal data can reduce dependence on a single data source and improve the stability and adaptability of the model in the face of different situations. By integrating different types of data, the model can analyze individuals from multiple dimensions and promote a comprehensive understanding of the individual's psychological state.

[0249] In one embodiment, the specific process of feature-level fusion includes: inputting feature sets from various data sources; reducing feature dimensions using methods such as PCA and ICA while retaining important cross-modal information; and outputting feature sets after dimensionality reduction, which are fused into a unified feature space.

[0250] In one embodiment, the specific process of decision-level fusion includes: inputting independent decision outputs of each model; combining the outputs of different models and assigning scores to each decision through a decision template; and outputting a final decision result that comprehensively considers the outputs of each model.

[0251] In one embodiment, the specific process of model-level fusion includes: inputting the above-mentioned data features; acquiring data features through different input layers, and then performing model-level fusion through a fusion layer; and outputting a model output result that integrates multiple data modalities.

[0252] Feature fusion provides a comprehensive perspective for psychological models by integrating data from different sources, making the models more accurate and effective in predicting and analyzing individual psychological states. This approach not only optimizes the performance of the model, but also enhances the model's ability to handle complex data.

[0253] 2.2.7 Transfer Learning

[0254] The embodiment of the present invention uses transfer learning to fine-tune the personal psychological model after initialization to further iteratively optimize and adapt to personal data. The fine-tuning strategy of the embodiment of the present invention is mainly divided into two points: the first point is layer freezing. The embodiment of the present invention freezes part of the layers of the personal psychological model after initialization, especially the layers close to the input, because they capture more general features, and then gradually releases more layers for fine-tuning, which is called staged thawing; the second point is learning rate adjustment, using a higher learning rate for the fine-tuned layer, and a lower learning rate or not updating the frozen layer. Such learning rate adjustment can use learning rate decay or cyclic learning rate. In some embodiments, the frozen layer may not be updated. The embodiment of the present invention inputs the data features after feature fusion (including micro-expressions, facial expressions, pupil changes and body language of video information, language, intonation, tone and rhythm pause of voice information, and body temperature, breathing rate, heart rate and human body detection of infrared information) into the personal psychological model for transfer learning to complete the iterative optimization of the model. Each time the user uses the model for psychological assessment or psychological therapy, further transfer learning can be performed, and such a model can become a unique psychological model of the individual.

[0255] Specifically, fine-tuning is the process of adjusting a pre-trained model to adapt to a new task in transfer learning. It allows the model to refine its parameters to better represent the psychological characteristics of a specific user, thereby improving the performance and accuracy of the model on a specific task.

[0256] In the context of psychological models, each individual’s data is unique, including their facial expressions, voice, and physiological signals. Through fine-tuning, the model can shift from general features to more personalized feature learning, thereby providing more personalized psychological assessments or treatment recommendations.

[0257] In addition to layer freezing and learning rate adjustment, fine-tuning can also include other strategies: data augmentation, which introduces more data variations during training to improve the generalization ability of the model; regularization techniques, which include, for example, random dropout or L2 regularization, to reduce overfitting and improve the generalization of the model; model pruning, which removes unimportant model parameters to make the model more concise and improve its performance on specific tasks.

[0258] Compared to training a model from scratch, transfer learning can achieve higher performance faster because it leverages knowledge pre-trained on related tasks. Transfer learning can leverage knowledge already learned by pre-trained models, reducing the need for large amounts of labeled data. Through transfer learning, the model can learn generalized features from a wider range of data, and then fine-tune it to optimize for specific user data, thereby improving the model's generalization ability and personalized adaptability.

[0259] Mental Model Generation Dialogue:

[0260] After the personal psychological model is created, it can take the language of the user during the conversation and the above-mentioned audio and video infrared data as input, and the personal psychological model can give corresponding answers. The personal psychological model can be required to comfort and empathize with the user through the prompt word mode, and can also give actions that the user can perform when necessary, such as providing the user with some solutions that can currently relieve emotions, such as listening to music, playing games, etc. At the same time, the personal psychological model can also judge the user's answer. The personal psychological model can be controlled by this method to ensure that after a continuous conversation, enough information can be obtained for the user to judge the mental health status.

[0261] Specifically, the above sufficient information mainly includes: language content, which is the content expressed by the user in the conversation, including the vocabulary used, sentence structure and emotions expressed, which can reveal the user's thoughts, emotional state and potential psychological needs. Voice features, which include voice features such as the user's tone, speaking speed, volume and pitch, which can reflect the user's emotional state and possible psychological pressure. Facial expressions and micro-expressions, the user's facial expressions and micro-expressions captured by video data can provide non-verbal emotional and psychological state information. Body language, the user's body movements and postures are also important indicators of emotional and psychological state. Pupil changes, pupil changes can reflect people's emotional changes and points of interest, which is an important dimension of psychological state analysis. Physiological data, physiological data captured by infrared and other technologies, such as body temperature, respiratory rate and heart rate, these data can reflect the user's physiological reactions and are closely related to the psychological state. User behavioral response: The user's response in the conversation, including the content of the answer, the choice of decision, etc., is also important information for evaluating the psychological state.

[0262] The information here mainly refers to the dialogue between users and their personal mental models, for example:

[0263] Personal psychological model: Hello, classmate. I am a psychological counselor. I am very happy to communicate with you today. What do you like to do in your daily life? Are you happy recently?

[0264] User: I like reading and I am having a very happy time recently. My parents are chefs.

[0265] Personal psychological model: Thank you for your self-introduction. First of all, the teacher would like to talk to you about your family life and your relationship with your parents. Can you tell me if you and your parents often chat together, if you often eat together, watch TV together or go to the park together, if you are unhappy, will you share it with your parents? Then tell me how you feel when you stay at home, whether you feel happy and relaxed or nervous and afraid.

[0266] User: Well, I often chat with my dad. We eat together, watch TV, and go to the park. I feel relaxed and happy at home.

[0267] In addition to the conversation information, there is also multimodal data during the conversation. When the personal psychological model determines the content of the communication and meets the information required by the psychological healing or assessment framework, the conversation will stop and the user will be informed that the healing or assessment is complete. This information will then be input into the personal psychological model. The output layer of the personal psychological model can output the following information: psychological state assessment, user psychological indicator score, key risk warning, risk intervention measures, and other psychological information.

[0268] Mental state assessment refers to the overall evaluation of the user's mental state provided by the personal psychological model based on the collected data. It does not involve the specific content of the conversation, but is a comprehensive evaluation based on information, including the evaluation of the user's self-confidence, social skills, family relationships, hobbies, academic pressure, etc. For example: As a high school student, you have shown considerable confidence in your independence and adaptability. You actively participate in school club activities and make new friends in them, showing good social skills. In terms of family, although you communicate less with your parents, you say that the family atmosphere is harmonious. You like sports, such as basketball and badminton, which helps you relieve stress after studying. Academically, you feel a certain amount of learning pressure and have a relatively clear vision for your future career planning, including considering working in the medical or military industry. Your vision and understanding show a certain positive attitude, but you may need some improvement in learning methods and learning habits.

[0269] User psychological index scoring refers to the personal psychological model scoring the user's psychological indicators, such as self-cognition and self-esteem, communication openness, interpersonal skills, etc. These psychological indicators are defined in the framework of psychological assessment and treatment. The personal psychological model uses the information in the user's conversation to derive the score of each indicator according to the indicator description of the framework.

[0270] Key risk warning means that a personal psychological model can determine whether a user has key risks and issue an identification warning by analyzing the information in the user's conversation, such as suicide risk, self-harm risk, drug abuse risk, etc.

[0271] Risk intervention means that when the personal psychological model determines that the user has a key risk and issues a warning, it will find the most suitable means based on the pre-stored intervention means and recommend risk intervention to avoid further expansion of the risk.

[0272] The remaining psychological information refers to the personal psychological model that will provide other psychological information based on the requirements and definitions of the psychological assessment and treatment framework, and has strong expansion capabilities.

[0273] The above information will be output and archived each time the user uses it. When the user uses the personal psychological model for treatment or evaluation many times, a time series of personal psychological files will be formed. The creation and application of personal psychological models provides a comprehensive and interactive way to evaluate and support personal mental health. This system combines the analysis of conversation, audio and video, near infrared, thermal imaging, high-speed photography data, and artificial intelligence-based processing technology to achieve in-depth personal psychological assessment and intervention.

[0274] The advantages of the above technical solution are:

[0275] By combining multimodal data, personal psychological models are able to provide a comprehensive mental health assessment that takes into account the user's behavior, emotions, and social interactions. This approach is able to capture subtle emotions and attitudes that users may not be able to express directly, thereby providing more personalized and in-depth support.

[0276] The personal psychological model can instantly analyze the conversation content and user performance to provide real-time mental health feedback. When potential risks or problems are detected, it can immediately intervene and provide intervention suggestions, such as recommending relaxing activities or professional psychological consultation.

[0277] Over time, the personal psychological model can collect and archive the user's psychological data to build a time-series mental health profile. This helps monitor long-term trends in personal mental health, identify potential problems, and support continued personal growth and development.

[0278] When key risks are identified, such as suicidal tendencies, self-harm or substance abuse, the model can provide timely warnings and recommend appropriate intervention measures based on the psychological expert system to reduce the occurrence of these risks.

[0279] By scoring and providing feedback on users' psychological indicators, personal psychological models can help users understand their own mental state, improve self-cognition, and enhance communication and interpersonal skills.

[0280] By using this technology, efficient and low-cost mental health assessment and intervention services can be provided to more people, especially in resource-limited areas or for those who have difficulty accessing traditional psychological counseling services.

[0281] Figure 4 is a functional block diagram of an artificial intelligence psychological assessment system based on multimodal input according to an embodiment of the present invention, such as Figure 4 As shown, an artificial intelligence psychological assessment system based on multimodal input, the system 200 includes:

[0282] A personal psychological model training module 210 is used to establish and train a personal psychological model for psychological assessment;

[0283] A dialogue module 220 is used to receive a psychological assessment request initiated by a user and to conduct a dialogue with the user through the personal psychological model;

[0284] The multimodal data acquisition module 230 is used to acquire the multimodal data of the user during the conversation with the user;

[0285] The mental health assessment module 240 is used to process and analyze the multimodal data of the user and conduct a mental health assessment on the user.

[0286] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0287] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0288] The units involved in the embodiments of the present invention may be implemented by software or hardware. The units described may also be arranged in a processor, and the names of these units do not constitute limitations on the units themselves in some cases.

[0289] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An artificial intelligence psychological assessment method based on multimodal input, characterized in that: The method comprises the following steps: S10: Build and train personal psychological models for psychological assessment; S20: receiving a psychological assessment request initiated by a user, and communicating with the user through the personal psychological model; S30: Acquiring multimodal data of the user during the conversation with the user; S40: Processing and analyzing the multimodal data of the user, and conducting a mental health assessment on the user; Step S10 specifically includes: S11: Optimizing the pre-trained large language model, and adapting the large language model to the context of psychological assessment by stepwise distillation; S12: Initialize the pre-trained large language model based on the user's basic information to obtain the first person psychological model; S13: communicating with the user through the first personal psychological model; S14: during the conversation, obtaining multimodal data of the user, wherein the multimodal data includes video data, audio signals, thermal imaging data, near infrared data, and high-speed photography data; S15: Processing the multimodal data; S16: using transfer learning to input the processed multimodal data into the first personal psychological model to perform training iterative optimization to obtain a second personal psychological model; Step S12 specifically includes the following steps: S121: Collect basic user information, and encode the basic user information into a processing format of a large language model, wherein the encoding method includes at least one of one-hot encoding, embedding vector, and combined features; S122: Load a large pre-trained language model based on the Transformer architecture from storage; S123: Adjusting parameters of the pre-trained large language model to obtain the first personal psychological model, which includes: Add one or more new neural network layers on top of the pre-trained large language model neural network; The encoded user basic information is used as the input of a pre-trained large language model with a new neural network layer added, combined with the input of each Transformer layer of the pre-trained large language model, and a set of adaptive weights is output; Combining the basic weights and adaptive weights of the pre-trained large language model, a new weight matrix is ​​determined by a hybrid weighted summation method to personalize the pre-trained large language model to obtain the first personal psychological model; wherein the hybrid parameter α of the hybrid weighted summation has a value range between 0 and 1; S124: Based on the user basic information, independently storing the first personal psychological model corresponding to the user; Step S15 specifically includes: Step S151: Processing the video data in the multimodal data; Step S152: Processing the audio signal in the multimodal data; Step S152 specifically includes: an audio preprocessing sub-step, a speech recognition sub-step, a tone and intonation analysis sub-step, and a rhythm pause analysis sub-step; The audio preprocessing sub-step comprises: Performing noise reduction processing on the audio signal; Segment the audio signal after the noise reduction process to obtain an audio frame after the noise reduction and segmentation process; Extracting spectrum features from the audio frames that have undergone noise reduction and segmentation processing, and normalizing the extracted spectrum features to obtain preprocessed audio signals; The speech recognition sub-step comprises: Taking the preprocessed audio signal as input, using a sequence-to-sequence model to perform speech recognition, and outputting a preliminary text sequence corresponding to the preprocessed audio signal; wherein the sequence-to-sequence model includes an encoder and a decoder with an attention mechanism, and introduces a reinforcement learning strategy, using the minimum word error rate as a reward function for model training; Taking the preprocessed audio signal and the preliminary text sequence as input, performing joint training and joint decoding in combination with the converter model, and outputting an optimized text sequence; The tone and intonation analysis sub-step includes: Extracting intonation patterns from the preprocessed audio signal using a pitch detection algorithm; Extracting key data features from the preprocessed audio signal using Mel frequency cepstral coefficients and linear prediction cepstral coefficients, and inputting the key data features into a pre-trained deep neural network to extract acoustic features; Analyzing the speaker's emotional state using a speech emotion recognition algorithm according to the intonation pattern and the acoustic features, and generating data features of tone and intonation; The rhythm pause analysis sub-step comprises: Pre-emphasize the pre-processed audio signal and output the pre-emphasized audio signal; Performing speech energy detection on the pre-emphasized audio signal, calculating the energy profile of the pre-emphasized audio signal, and identifying areas where the energy is lower than a preset threshold, marking them as silence or pause; Performing spectral entropy analysis on the pre-emphasized audio signal to measure the entropy value of the spectrum to distinguish between speech segments and non-speech segments; Extracting time domain features from the pre-emphasized audio signal, wherein the time domain features include a zero crossing rate ZCR; Extracting frequency domain features from the pre-emphasized audio signal, wherein the frequency domain features include spectrum centroid and spectrum flatness, and are used to analyze spectrum characteristics of the audio signal; Based on the extracted time domain features and frequency domain features, data features of rhythm pauses are generated for use in evaluating psychological states.

2. The artificial intelligence psychological assessment method based on multimodal input according to claim 1, characterized in that: Step S151 specifically includes: a video preprocessing step, a body movement data extraction step, a facial expression analysis step, and a pupil change detection step; The video preprocessing step specifically includes: extracting frames from the video data to decompose the video data into separate video frames; adjusting the resolution of the video frames; normalizing the video data to standardize the brightness and contrast of the video frames to obtain preprocessed video frames; The limb motion data extraction step specifically includes: using a human posture estimation algorithm to detect and track key points of the human body in the preprocessed video frame, and identifying specific limb motions and behavior patterns by analyzing the movement of key points of the human body over time, and generating limb motion data features; The facial expression analysis step specifically includes: using a facial detection algorithm to identify the facial area in the preprocessed video frame, and using a facial feature point detection algorithm to track key feature points of the face to obtain facial expression data features of the current user; Among them, the pupil change detection step specifically includes: using an eye area positioning algorithm to determine the points related to the eyes among the facial feature points in the preprocessed video frame, and using a pupil positioning algorithm to determine the exact position of the pupil, and then performing pupil size change analysis to generate pupil change data features.

3. The artificial intelligence psychological assessment method based on multimodal input according to claim 1, characterized in that: Step S15 specifically includes step S153: processing the thermal imaging data in the multimodal data; The step S153 comprises: Preprocessing of thermal imaging data; Using a thermal imaging processing algorithm to locate the thermal area of ​​the human body for the preprocessed thermal imaging data, and extracting body temperature data from the thermal area; For the pre-processed thermal imaging data, based on the dynamic heat source, the thermal image changes of the mouth, nose and chest area are scanned to monitor the temperature changes caused by breathing. The temperature changes of the mouth, nose and chest as well as the temperature changes of the gas produced by breathing are analyzed through time series to calculate the respiratory frequency. Heart rate monitoring by identifying microcirculatory changes in the facial area from preprocessed thermal imaging data; The dynamic heat sources in the preprocessed thermal imaging data are analyzed, the human body is recognized using a motion detection algorithm, and morphological operations are performed to identify the human body contour.

4. The artificial intelligence psychological assessment method based on multimodal input according to claim 1, characterized in that: Step S15 specifically includes step S154: processing the near infrared data in the multimodal data; The step S154 comprises: Based on the Lambert-Beer law, the concentration change of hemoglobin is calculated according to the known incident light intensity and the captured outgoing light intensity; By analyzing the changes in hemoglobin concentration, the activity data characteristics of different areas of the human frontal cerebral cortex are obtained.

5. The artificial intelligence psychological assessment method based on multimodal input according to claim 1, characterized in that: Step S15 specifically includes step S155: processing the high-speed photography data in the multimodal data; The step S155 specifically includes: Acquire black and white high-speed photography data; Performing histogram equalization processing on the black-and-white high-speed photography data to obtain a black-and-white image, and performing standardization processing on pixel values ​​of the black-and-white image; A multi-task convolutional neural network model is used to obtain facial features from high-speed photography data images after histogram equalization and standardization, and multiple muscle features of the human face are extracted. The facial features extracted from the multi-task convolutional neural network model are input into a three-dimensional convolutional neural network to perform micro-expression recognition on time series data, and the output is a multi-dimensional feature tensor to obtain micro-expression data; Using the pupil-corneal reflection method, the eyes are illuminated by infrared rays and the infrared light reflected from the cornea and retina is collected. The pupil positioning and light spot detection algorithm is used to obtain the pupil position and the position of the light spot on the cornea from the infrared reflection image to determine the line of sight of the eye and obtain eye movement data.

6. The artificial intelligence psychological assessment method based on multimodal input according to claim 1, characterized in that: Step S15 specifically includes step S156: using transfer learning to input the processed multimodal data into the first personal psychological model to perform training iterative optimization, thereby obtaining a second personal psychological model; Step S156 includes the following steps: performing transfer learning on a pre-trained base psychological model, the base psychological model serving as a first person psychological model; Implementing a hierarchical freezing rule on preset layer data of the first personal psychological model, the preset layer data including a bottom layer, a top layer, and a preset middle layer of the first personal psychological model; Using a phased unfreezing method, gradually unfreeze the preset middle layer and top layer of the first personal psychological model; For the unfrozen preset middle layer and top layer, fine-tune the corresponding layer parameters using a first learning rate, and fine-tune the layer parameters of the unfrozen bottom layer of the first personal psychological model using a second learning rate, wherein the first learning rate is greater than the second learning rate, to obtain the first personal psychological model after the model parameters are fine-tuned; Fusion of micro-expressions, facial expressions, pupil changes and body language features from video data; language content, intonation, tone and rhythm pause features from voice data; and body temperature, breathing rate, heart rate and human movement features from infrared data; The data features after feature fusion are used for transfer learning, and the first person psychological model after fine-tuning the model parameters is iteratively optimized to obtain a second person psychological model.

7. An artificial intelligence psychological assessment system based on multimodal input, characterized in that: The system comprises: Personal psychological model training module, used to establish and train personal psychological models for psychological assessment; A dialogue module, used for receiving a psychological assessment request initiated by a user, and conducting a dialogue with the user through the personal psychological model; A multimodal data acquisition module, used to acquire multimodal data of the user during the conversation with the user; A mental health assessment module, used to process and analyze the multimodal data of the user and conduct a mental health assessment on the user; The personal psychological model training module is specifically used to perform the following steps: S11: Optimizing the pre-trained large language model, and adapting the large language model to the context of psychological assessment by stepwise distillation; S12: Initialize the pre-trained large language model based on the user's basic information to obtain the first person psychological model; S13: communicating with the user through the first personal psychological model; S14: during the conversation, obtaining multimodal data of the user, wherein the multimodal data includes video data, audio signals, thermal imaging data, near infrared data, and high-speed photography data; S15: Processing the multimodal data; S16: using transfer learning to input the processed multimodal data into the first personal psychological model to perform training iterative optimization to obtain a second personal psychological model; Step S12 specifically includes the following steps: S121: Collect basic user information, and encode the basic user information into a processing format of a large language model, wherein the encoding method includes at least one of one-hot encoding, embedding vector, and combined features; S122: Load a large pre-trained language model based on the Transformer architecture from storage; S123: Adjusting parameters of the pre-trained large language model to obtain the first personal psychological model, which includes: Add one or more new neural network layers on top of the pre-trained large language model neural network; The encoded user basic information is used as the input of a pre-trained large language model with a new neural network layer added, combined with the input of each Transformer layer of the pre-trained large language model, and a set of adaptive weights is output; Combining the basic weights and adaptive weights of the pre-trained large language model, a new weight matrix is ​​determined by a hybrid weighted summation method to personalize the pre-trained large language model to obtain the first personal psychological model; wherein the hybrid parameter α of the hybrid weighted summation has a value range between 0 and 1; S124: Based on the user basic information, independently storing the first personal psychological model corresponding to the user; Step S15 specifically includes: Step S151: Processing the video data in the multimodal data; Step S152: Processing the audio signal in the multimodal data; Step S152 specifically includes: an audio preprocessing sub-step, a speech recognition sub-step, a tone and intonation analysis sub-step, and a rhythm pause analysis sub-step; The audio preprocessing sub-step comprises: Performing noise reduction processing on the audio signal; Segment the audio signal after the noise reduction process to obtain an audio frame after the noise reduction and segmentation process; Extracting spectrum features from the audio frames that have undergone noise reduction and segmentation processing, and normalizing the extracted spectrum features to obtain preprocessed audio signals; The speech recognition sub-step comprises: Taking the preprocessed audio signal as input, using a sequence-to-sequence model to perform speech recognition, and outputting a preliminary text sequence corresponding to the preprocessed audio signal; wherein the sequence-to-sequence model includes an encoder and a decoder with an attention mechanism, and introduces a reinforcement learning strategy, using the minimum word error rate as a reward function for model training; Taking the preprocessed audio signal and the preliminary text sequence as input, performing joint training and joint decoding in combination with the converter model, and outputting an optimized text sequence; The tone and intonation analysis sub-step includes: Extracting intonation patterns from the preprocessed audio signal using a pitch detection algorithm; Extracting key data features from the preprocessed audio signal using Mel frequency cepstral coefficients and linear prediction cepstral coefficients, and inputting the key data features into a pre-trained deep neural network to extract acoustic features; Analyzing the speaker's emotional state using a speech emotion recognition algorithm according to the intonation pattern and the acoustic features, and generating data features of tone and intonation; The rhythm pause analysis sub-step comprises: Pre-emphasize the pre-processed audio signal and output the pre-emphasized audio signal; Performing speech energy detection on the pre-emphasized audio signal, calculating the energy profile of the pre-emphasized audio signal, and identifying areas where the energy is lower than a preset threshold, marking them as silence or pause; Performing spectral entropy analysis on the pre-emphasized audio signal to measure the entropy value of the spectrum to distinguish between speech segments and non-speech segments; Extracting time domain features from the pre-emphasized audio signal, wherein the time domain features include a zero crossing rate ZCR; Extracting frequency domain features from the pre-emphasized audio signal, wherein the frequency domain features include spectrum centroid and spectrum flatness, and are used to analyze spectrum characteristics of the audio signal; Based on the extracted time domain features and frequency domain features, data features of rhythm pauses are generated for use in evaluating psychological states.

Images