A multi-modal sensor based feature image fusion psychological assessment system

Abstract

This invention discloses a feature profile fusion psychological assessment system based on multimodal sensors, including an interaction module, a multimodal data acquisition module, a data processing module, and an assessment result output module. The interaction module generates questionnaires and records user responses. The multimodal data acquisition module collects relevant psychological activity data and sleep indicators from the user's response process. The data processing module processes and analyzes the collected data and sleep indicators, constructs a user feature profile, and fuses the user's feature profile with the question-and-answer data to analyze the user's psychological state. The beneficial effects of this invention are: by using individualized baseline standardization and sleep state correction, it reduces the interference of physical differences and fatigue factors on the assessment; by introducing the signal quality coefficient Qi into the scoring process, it automatically reduces weights or re-acquires data when the acquisition quality is poor, improving the system's robustness.

Description

Technical Field

[0001] This invention relates to the technical field of psychological assessment, specifically to a psychological assessment system based on feature profile fusion using multimodal sensors. Background Technology

[0002] In existing technologies, psychological assessments of the elderly typically rely on questionnaires or simple self-reported conversations, which are easily affected by factors such as social expectations, avoidance of expression, memory bias, hearing impairment, cognitive impairment, and daily fatigue, resulting in large fluctuations and insufficient reliability of assessment results. While wearable devices can collect information such as heart rate, electrocardiogram, blood oxygen, and sleep, they lack an assessment mechanism that is synchronized with the question-and-answer process. They cannot verify and close the loop of consistency between "response content" and "physiological response," making it difficult to achieve continuous assessment and risk classification reminders for home-based elderly care scenarios.

[0003] Existing psychological assessment techniques are mainly divided into two categories, but both have significant limitations. The first category is traditional self-report questionnaire assessments, including standardized tools such as the Geriatric Depression Scale, PHQ-9, and SAS. Although these have mature norm references and clinical validation, they rely on the subjective reports of the test takers and are easily affected by factors such as social expectation effects, memory bias, avoidance defenses, and cognitive impairments, leading to insufficient reliability of the results. Especially for the elderly, hearing loss, slow reaction, and difficulty in comprehension further weaken the applicability of the questionnaires. Moreover, traditional questionnaires lack real-time monitoring of the answering process and cannot identify "hypocritical" or perfunctory behaviors. The second category is emerging biosensor assessments, which use wearable devices to collect physiological indicators such as heart rate, heart rate variability, and skin conductance to directly infer psychological state. Although they can capture objective physiological responses, they ignore individual differences in physical condition (e.g., athletes have lower resting heart rates, and patients with chronic diseases have higher baseline arousal), resulting in a higher misjudgment rate. At the same time, purely physiological assessments lack standardized norms, making it difficult to connect the results with clinical diagnoses, and they cannot distinguish between physiological stress and psychological disorders.

[0004] Therefore, we designed a feature profile fusion psychological assessment system based on multimodal sensors to solve the above problems. Summary of the Invention

[0005] Purpose of the invention: To provide a feature profile fusion psychological assessment system based on multimodal sensors to solve the above-mentioned problems existing in the prior art.

[0006] Technical Solution: A feature profile fusion psychological assessment system based on multimodal sensors, comprising an interaction module, a multimodal data acquisition module, a data processing module, and an assessment result output module. The interaction module generates questionnaires and records user responses. The multimodal data acquisition module collects relevant psychological activity data and sleep indicators related to the user's answering process. The data processing module processes and analyzes the collected data and sleep indicators, constructs a user feature profile, merges the user's feature profile with the question-and-answer data, and analyzes the user's psychological state. The assessment result output module outputs the questionnaire assessment results and pushes risk warnings to the client.

[0007] Preferably, the multimodal data acquisition module includes a voice acquisition unit, an interaction acquisition unit, a physiological signal acquisition unit, and a sleep data acquisition unit; the voice acquisition unit acquires the user's voice signals during the question-and-answer process via a microphone; the interaction acquisition unit is used to acquire the user's interaction data during the question-and-answer process, including the reaction time, answer time, pause ratio, and number of interruptions for each questionnaire question; the physiological signal acquisition unit and the sleep data acquisition unit are integrated into the wearable device, the physiological signal acquisition unit is used to acquire the user's physiological signals during the question-and-answer process, including heart rate, respiration, electrocardiogram, and blood oxygen; the sleep data acquisition unit is used to acquire the user's sleep indicators before the assessment, including sleep quality, sleep duration, and sleep stage information.

[0008] Preferably, the data processing module includes a data preprocessing unit, a time-series alignment unit, an individualized baseline construction unit, a feature extraction unit, a feature profile generation unit, a credibility calculation unit, and a verification unit. The preprocessing unit cleans and denoises the data collected by the multimodal data acquisition module. The time-series alignment unit aligns the answering time period of each questionnaire question with the corresponding physiological signals and interaction data to establish a unified timeline, and constructs an advance baseline window, an answer window, and a post-question reply window for each question and answer data. The individualized baseline construction unit calculates the mean and standard of each physiological characteristic based on the subject's historical resting state data and physiological data during casual conversation. The system establishes an individualized baseline model. The feature extraction unit standardizes the physiological features corresponding to each questionnaire item based on the mean and standard deviation of physiological features. The feature profile generation unit deeply integrates the standardized physiological features, interaction features, and questionnaire response features, using an LSTM-FCNN hybrid model to concatenate the feature vectors of the three single-dimensional profiles and mine temporal correlations to output the user's psychological profile. The credibility calculation unit evaluates the credibility of the question-and-answer data for each questionnaire item based on interaction data and standardized physiological features, corrects the credibility of each question-and-answer data, and obtains the user's total psychological state score and risk level. The verification unit verifies the credibility of question-and-answer data with low credibility by re-asking the user with relevant questions and combining corresponding physiological signals, interaction data, and sleep indicators.

[0009] Preferably, the standardized expression of the physiological characteristic is: , in, z i,k For the first i Question No. k Standardized values ​​of physiological characteristics x i,k For the first i Question No. k A physiological characteristic, m k For the first k Individualized baseline mean of each physiological characteristic s k For the first k Individualized baseline standard deviation of each physiological characteristic It is a very small constant.

[0010] Preferably, the expression for the total psychological state score is: , in, M The total score represents the psychological state during the conversation. For the first i The weight of the question For the first i The credibility score of the question. For credibility correction function, For the first i The original psychological score of the question.

[0011] Preferably, the data processing module is further provided with a storage unit for storing the data collected by the multimodal data acquisition module.

[0012] Preferably, it also includes a data security module to encrypt and store all collected data and set access control to allow only authorized personnel to access the data through a specific secure channel.

[0013] The beneficial effects of this invention are: By standardizing individualized baselines and correcting for sleep states, this invention reduces the interference of physical differences and fatigue factors on the assessment; and by obtaining reliability through a consistency evaluation of "negativity of response content – ​​physiological arousal response". C i For questionable or unstable answers, a review is triggered to reduce bias caused by relying solely on self-reporting; combining wearable continuous data collection and proactive question answering, trends and tiered intervention suggestions are output to form a closed-loop management system; and the signal quality coefficient is... Q i A scoring process is introduced to automatically reduce the weight or re-collect data when the data collection quality is poor, thereby improving the robustness of the system. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the structure of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Specific Implementation

[0016] As attached Figure 1 As shown in this embodiment, a feature profile fusion psychological assessment system based on multimodal sensors includes an interaction module, a multimodal data acquisition module, a data processing module, and an assessment result output module. The interactive module is used to generate questionnaires and record users' answers. The multimodal data acquisition module is used to collect relevant psychological activity data and sleep indicators of users during the question-answering process; The data processing module is used to process and analyze the collected data and sleep indicators, construct user feature profiles, integrate user feature profiles with question and answer data, and analyze user psychological state. The assessment result output module is used to output the assessment results of the questionnaire and push risk warnings to the client.

[0017] The interaction module includes a speaker, a communication module, and a display screen. When in use, the data processing module establishes communication with the user's wearable device via Bluetooth. The wearable device includes a bracelet or a finger clip, etc. In this invention, a bracelet is used as the wearing device.

[0018] The interaction module outputs a psychological assessment questionnaire to the user and displays it on the screen. Users can also choose to have the questions read aloud via a speaker. The user views the questionnaire on the screen and responds verbally. The multimodal data acquisition module collects data from the responses, while the microphone array of the voice acquisition unit captures the user's voice responses as question-and-answer data. Simultaneously, the interaction acquisition unit records interaction data, including reaction time, response time, pause ratio, and number of interruptions. The wearable device synchronously collects the user's physiological signals, including at least heart rate, respiration, electrocardiogram, and blood oxygen saturation, as well as sleep-related indicators such as sleep quality, sleep duration, and sleep stage information. The processor in the data processing module receives and processes the collected question-and-answer data, interaction data, physiological signals, and sleep indicators, storing the collected data in memory. The collected data is processed and analyzed to construct a user profile, which is then fused with the user's question-and-answer data to determine the user's psychological state.

[0019] Meanwhile, this assessment questionnaire can be activated through a preset schedule or triggering conditions, such as a significant decrease in sleep duration, a deep sleep rate below the individual threshold, a decrease in activity level in the past three days, or when the elderly person actively expresses discomfort.

[0020] The multimodal data acquisition module includes a voice acquisition unit, an interactive acquisition unit, a physiological signal acquisition unit, and a sleep data acquisition unit; The voice acquisition unit collects the user's voice signals during the question-answering process via a microphone; the interaction acquisition unit is used to collect the user's interaction data during the question-answering process, including the reaction time, answer time, pause ratio, and number of interruptions for each questionnaire question.

[0021] The physiological signal acquisition unit and the sleep data acquisition unit are integrated into the wearable device. The physiological signal acquisition unit is used to collect physiological signals during the user's answering process, including heart rate, respiration, electrocardiogram and blood oxygen; the sleep data acquisition unit is used to collect the user's sleep indicators before the assessment, including sleep quality, sleep duration and sleep stage information.

[0022] The data processing module includes a data preprocessing unit, a time-series alignment unit, an individualized baseline construction unit, a feature extraction unit, a feature profile generation unit, a credibility calculation unit, and a verification unit. The preprocessing unit cleans and denoises the data acquired by the multimodal data acquisition module; The time alignment unit aligns the answering time period of each questionnaire question with the physiological signals and interaction data of the corresponding time period to establish a unified time axis, and constructs an advance baseline window, an answer window, and a post-question reply window for each question and answer data. The individualized baseline construction unit establishes an individualized baseline model by statistically analyzing the mean and standard deviation of each physiological characteristic based on the subject's historical resting state data and physiological data during casual conversation. The feature extraction unit standardizes the physiological features corresponding to each questionnaire item based on the mean and standard deviation of the physiological features. The feature profile generation unit deeply integrates standardized physiological features, interaction features, and questionnaire response features. It uses an LSTM-FCNN hybrid model to concatenate the feature vectors of the three single-dimensional profiles and mine temporal correlations to output the user's psychological feature profile. Psychological profiling can comprehensively and meticulously reflect a user's psychological characteristics and state. By deeply mining and fusing multiple standardized features, the LSTM-FCNN hybrid model leverages the advantages of Long Short-Term Memory (LSTM) networks in processing sequential data and the efficiency of Fully Convolutional Neural Networks (FCNN) in feature extraction.

[0023] In the process of concatenating feature vectors, the model organically combines physiological features, interaction features, and questionnaire response features, allowing information from different dimensions to complement and corroborate each other. For example, heart rate changes in physiological features may be related to a user's anxiety while answering questions, while answering speed and the number of pauses in interaction features can reflect a user's thought process and decision-making difficulty. By concatenating these features, the model can more comprehensively capture the user's psychological activities.

[0024] Throughout the assessment process, a user's psychological state may dynamically change with time and the content of the questions. The LSTM-FCNN hybrid model can identify the underlying patterns and regularities in these changes, thus more accurately characterizing the user's psychological traits. For example, when certain types of questions appear, the user's physiological and interaction characteristics may change synchronously, potentially indicating a specific psychological response to these types of questions.

[0025] The generated psychological profiles can provide important information for subsequent psychological assessments and analyses. They not only help professionals gain a deeper understanding of users' psychological states but can also be used to develop personalized psychological intervention and treatment plans. Furthermore, psychological profiles can serve as an objective indicator for evaluating users' psychological stability and adaptability in different situations.

[0026] The credibility calculation unit evaluates the credibility of the question and answer data for each questionnaire question based on the interaction data and standardized physiological characteristics, corrects the credibility of each question and answer data, and obtains the total score of the user's psychological state and risk level in the session. The verification unit is used to verify question-and-answer data with low credibility. The verification unit will ask the user relevant questions again and combine them with the corresponding physiological signals, interaction data and sleep indicators to re-verify the credibility of the question-and-answer data.

[0027] After data collection is complete, the processor processes the data, establishes a timeline, and precisely aligns the timestamps of the question-and-answer data with the timestamps of the physiological signals to ensure their synchronization on the timeline. Subsequently, based on the aligned data, an analysis window is constructed, with a pre-baseline window, an answer window, and a post-question reply window set up for each question-and-answer data point to comprehensively capture the user's physiological changes during the question-and-answer process.

[0028] The processor uses historical data to build an individualized baseline model for each user, collects resting window data in non-evaluation states, and collects conversation window data during ordinary casual conversations, calculating the mean and standard deviation of each feature respectively. m k , sk These baseline models reflect the mean and standard deviation of users' physiological characteristics under normal conditions. Based on the individualized baseline model, the processor standardizes the currently collected physiological characteristics to eliminate the influence of individual differences on the assessment results.

[0029] The interactive data is collected by the interactive acquisition unit, including reaction time, response time, pause ratio, and number of interruptions; the physiological signals are collected by the wearable device, including at least heart rate, respiration, electrocardiogram, and blood oxygen; the sleep-related indicators are collected by the wearable device, including sleep quality, sleep duration, and sleep stage information.

[0030] The collected data is transmitted to the processor in real time. The processor in the data processing module analyzes and processes the data. First, it cleans the received data to remove outliers caused by equipment malfunctions or signal interference, ensuring the accuracy and reliability of the data. Next, the processor uses a preset algorithm to perform in-depth analysis of the cleaned data, extracting characteristic indicators closely related to the psychological state of the elderly. After extracting the characteristic indicators, the processor further performs a comprehensive evaluation based on these indicators. For each question-and-answer data point, the processor first uses a signal quality assessment algorithm, combining the stability and clarity of physiological signals, to map a signal quality coefficient, which is used to measure the reliability of the data point. The processor calculates a raw psychological score based on the question-and-answer content. This score reflects the degree of negativity in the user's response and is normalized to ensure comparability.

[0031] After obtaining the initial psychological score, the processor further calculates a credibility score, based on the consistency between the degree of negativity and the physiological arousal index. This score is combined with a signal quality coefficient and a sleep index correction term to comprehensively assess the credibility of each question and answer. The credibility score reflects the authenticity and reliability of the user's response, providing a basis for subsequent review and risk warnings.

[0032] The processor also calculates the user's total psychological state score based on preset weighting and credibility correction functions. This total score combines the weight of each question and answer, credibility score, and raw psychological score, providing a comprehensive reflection of the user's psychological state. Furthermore, based on the total psychological state score, the processor categorizes users into risk levels to promptly push risk alerts to the client, providing strong support for continuous assessment and risk grading reminders in home-based elderly care scenarios.

[0033] The question bank for this psychological assessment questionnaire is stored in the data processing module's memory. Each time a questionnaire is output, 8-15 questions covering different psychological dimensions are selected from the question bank, primarily using closed-ended questions. The psychological dimensions in the questionnaire include: low mood / interest, anxiety and worry, loneliness support, fatigue and sleep-related emotions, and helplessness and hopelessness risk warnings.

[0034] The interactive module broadcasts the first [item] in sequence. i Questions and record their weights. w i It can be preset or adaptive. For example, the weight of the "helpless and hopeless" dimension is higher than that of other dimensions.

[0035] For each question and answer data point, construct an advance baseline window, an answer window, and a post-question reply window.

[0036] For the first i Question definition window: Baseline window before question W pre 20 seconds; Answer window W ans Maximum 60 seconds; Restore window after question W post 20 seconds.

[0037] Baseline window before the question W pre Internally, the processor primarily collects and analyzes the user's physiological signal data before they hear the question, using this as a reference for the user's baseline physiological state. (Answer window) W ans This refers to the time period during which the user actually answers the question. During this period, the processor focuses on capturing changes in the user's physiological responses, such as increased heart rate and rapid breathing. These changes may be closely related to the user's psychological state. (Post-question recovery window) W post This is used to observe the recovery of physiological signals after the user answers a question, in order to assess whether the user's psychological state gradually returns to stability.

[0038] After constructing the analysis windows, the processor further extracts the physiological characteristics and their changes within each window. These characteristics include, but are not limited to, heart rate variability, respiratory rate, electrocardiogram waveform characteristics, and blood oxygen saturation. By analyzing these characteristics, the processor can more accurately capture the user's physiological changes during the question-and-answer process.

[0039] Physiological features were extracted from each item in the psychological questionnaire, including but not limited to: Heart rate characteristics: mean, maximum, and slope of heart rate in the response window; the difference ΔHR between the response window and the window before the question.

[0040] Respiratory characteristics: mean respiratory rate, variability; difference from the pre-examination window ΔRR.

[0041] ECG (HRV) characteristics: RMSSD and SDNN are obtained from the ECGR-R interval; and the difference between the ECGR-R interval and the window value ΔHRV.

[0042] Blood oxygenation characteristics: mean SpO2, minimum SpO2, and decrease ΔSpO2.

[0043] Sleep conversation-level characteristics (from last night): total sleep duration, deep sleep percentage, light sleep percentage, core sleep percentage, sleep quality score, and number of nighttime awakenings (if provided by the device).

[0044] Speech behavior characteristics: speech rate, pause ratio, volume variation, fundamental frequency fluctuation, and reaction time (RT).

[0045] Text semantic features: emotional polarity, density of negative words, keywords of helplessness or hopelessness, used only for risk warning and assessment, not for medical diagnosis.

[0046] For the first i Question No. k Physiological characteristics x i,k Standardization process: , in, z i,k For the first i Question No. k Standardized values ​​of physiological characteristics x i,k For the first i Question No. k A physiological characteristic, m k For the first k Individualized baseline mean of each physiological characteristic s k For the first k Individualized baseline standard deviation of each physiological characteristic It is a very small constant.

[0047] Physiological arousal index is calculated based on standardized physiological characteristics: , in, Physiological arousal index For the first k The weighting coefficients for each physiological characteristic can be set based on experience, such as assigning different coefficients to ΔHR, ΔRR, ΔHRV, and ΔSpO2. A max This is the upper limit cutoff value for the physiological arousal index, used to suppress the influence of outliers.

[0048] Calculate the signal quality coefficient for each question. Q i For example, the confidence level of ECGR peak detection, the proportion of effective PPG samples, the proportion of motion artifacts, and the availability of respiratory waveforms are mapped to 0-1. Q i < Q min When the value is 0.5, the system will prompt for resampling or the question will enter the review process.

[0049] Calculate the raw psychological score based on the question-and-answer data. S i The scoring can be done using a "keyword + degree adverb + negation structure" rule or a lightweight semantic classification model. S i A score of 0-4 can be given, with 0 indicating none, 1 indicating mild, 2 indicating moderate, 3 indicating moderate, and 4 indicating severe.

[0050] Will Si Normalization yields the degree of negativity: , in, For negative degree, The maximum value of the original psychological score is 4.

[0051] Construct fatigue correction parameters based on sleep characteristics , , in, F deep Sleep fatigue correction factor T deep For sleep duration, R deep For the proportion of deep sleep, T ref , R ref For individual reference values; These are the corrected weighting coefficients for sleep duration and the proportion of deep sleep, respectively.

[0052] Adjust the wake-up index: , in, No. i The revised physiological arousal index For the first i Physiological arousal index A max This is the upper limit cutoff value for the physiological arousal index.

[0053] The credibility score for each question and answer is calculated based on the consistency between the degree of negativity and the adjusted physiological arousal index. C i Consistency can be measured by calculating the correlation coefficient or the degree of difference between the two, such as using the Pearson correlation coefficient or the absolute difference value. When the degree of negativity is highly consistent with the physiological arousal index, it indicates that the user's answer is consistent with their physiological response and has high credibility; conversely, it has low credibility.

[0054] Combined with signal quality coefficient Q i With sleep index correction item F sleep The credibility score for each question and answer. C i Further corrections are needed. The signal quality coefficient is used to reflect the reliability and stability of physiological signals. Q iA lower score indicates that physiological signals may be interfered with or collected inaccurately, and the credibility score should be reduced in this case. The sleep index correction item takes into account the impact of the user's sleep state on the psychological assessment. When the user is sleep-deprived or has poor sleep quality, their psychological state may be affected, leading to a decrease in the credibility of the answer.

[0055] Based on the revised credibility score C i The system filters and verifies each question-and-answer data entry. For entries with low credibility, the system can automatically trigger a verification process, such as re-collecting physiological signals, prompting the user to answer again, or having a professional conduct a manual verification. Simultaneously, the system can push different levels of risk alerts to the client based on the credibility score, enabling timely intervention.

[0056] Define expected wake-up: , in, The expected arousal index for question i. The coefficient representing the degree of negativity on desired arousal. The degree of negativity of question i. The base offset to be woken up.

[0057] Consistency mismatch: , Credibility rating: , in, This is the attenuation coefficient for consistency mismatch.

[0058] when C i < C low If the confidence level is 0.4, the review strategy is triggered, and the question is re-asked once using the same dimension and different questions. If the re-asked question is still low confidence, the question is treated as low confidence and marked "Manual review recommended" in the report.

[0059] The system calculates the user's total psychological state score based on the weight of each question and answer, the adjusted credibility score, and the original psychological score. This total score comprehensively reflects the user's psychological state level, providing strong support for continuous assessment and risk grading alerts in home-based elderly care scenarios. Furthermore, the system can categorize users' risk levels based on their total psychological state score, such as low risk, medium risk, and high risk, allowing for different intervention measures to be implemented for users at different risk levels.

[0060] Calculate the total score of psychological state: , in, M The total score represents the psychological state during the conversation. For the first i The weight of the question For the first i The credibility score of the question. For credibility correction function, For the first i The original psychological score of the question. , .

[0061] when C i <C low This question will not be included in the calculation and will be transferred to the review process.

[0062] Example of grading: M < T 1 :normal; T 1 ≤ M < T 2 Pay attention (it is recommended to review and observe the trend the following day); M ≥ T 2 Or high-risk semantic features may appear: high risk (triggering immediate review and push notification).

[0063] The output includes: each question S i , C i Total Score M Level, trends over the past 7 / 30 days, and a list of "low-reliability questions".

[0064] The output will be automatically generated from the raw data collected by the interactive acquisition unit, the voice acquisition unit, and the wearable device. The "List of Low-Confidence Questions" must be labeled with specific question numbers and their corresponding information. C i The system will dynamically plot the psychological state fluctuation curve based on data from the past 7 and 30 days. If three consecutive readings of "Attention" or one reading of "High Risk" are detected, a manual intervention warning will be triggered. All data fields are transmitted using encryption to ensure user privacy and security.

[0065] For questions with low credibility, the interaction module switches to shorter, more specific, or no-questions, or asks the question again when the elderly are more relaxed; and pushes risk level, trend, credibility prompts, and suggestions to the family member or caregiver.

[0066] It also includes a data security module that encrypts and stores all collected data and sets access control, allowing only authorized personnel to access the data through a specific secure channel.

[0067] During data transmission, advanced encryption algorithms are used to encrypt the data, preventing it from being stolen or tampered with. The system also regularly backs up the data to prevent loss or corruption. The data security module also has a data auditing function, recording all data access and operation activities for tracing and investigation in the event of a security incident. Furthermore, the system continuously optimizes its data security strategy based on user usage and feedback, improving data security and reliability. For sensitive data involving user privacy, such as physiological signal data and psychological assessment results, the system employs even stricter security measures to ensure that users' privacy rights are not violated.

[0068] Working principle: This system works in conjunction with wearable data collection devices to assess the psychological state of elderly users through a pre-set psychological evaluation questionnaire. The interactive module reads the questionnaire questions sequentially and records the user's answers and corresponding physiological signal data. The wearable device collects the user's physiological signals in real time, such as heart rate, respiratory rate, electrocardiogram waveform characteristics, and blood oxygen saturation, and transmits this data to the processor for analysis. The processor processes each question-and-answer data according to a pre-set analysis window, extracting physiological characteristics and their changes, and calculating a physiological arousal index. Simultaneously, the processor calculates a raw psychological score based on the question-and-answer data and corrects each question-and-answer data by incorporating signal quality assessment and credibility scoring, obtaining the user's total psychological state score and risk level for the session. The processor generates a user profile based on the processed data and merges it with the total psychological score calculated from the question-and-answer data to analyze the user's psychological state. Based on the total psychological state score and risk level, the system pushes corresponding risk warnings and intervention suggestions to the client to ensure timely measures are taken to protect the mental health of elderly users. Furthermore, the system also has a data security module to ensure the security and privacy of user data.

[0069] The questionnaire used in this invention is shown in Table 1 below: Table 1. Geriatric Depression Scale

[0070] The scores for each dimension are shown in Table 2 below: Table 2 Scores for each dimension

[0071] During implementation, the interactive module guides elderly users to complete the questionnaire using gentle voice prompts, while wearable devices continuously collect physiological signals. After each question is answered, the system immediately preprocesses the collected physiological data, including filtering and noise reduction, and outlier removal, to ensure data quality. For each question, the system calculates the corresponding physiological arousal index based on physiological signal characteristics, and simultaneously calculates the raw psychological score based on the user's answers. During the calculation process, the system dynamically adjusts weighting coefficients; for example, for questions involving negative emotions, the weight of the physiological arousal index is appropriately increased to more accurately reflect the user's psychological state. After all questions are completed, the system calculates the user's total psychological state score based on the weight of each question, credibility score, and raw psychological score, and determines the user's risk level according to a preset grading standard. The system also generates a detailed assessment report, including the score for each question, credibility score, total psychological state score, risk level, and recent trends in psychological state changes. The assessment report is presented in intuitive charts and text for easy understanding by the user and their family. For elderly users at risk of depression, the system will provide corresponding intervention suggestions based on the risk level, such as recommending increased social activities, physical exercise, and seeking professional psychological counseling. Simultaneously, the system will push the assessment results and intervention suggestions to family members or caregivers so that they can promptly understand the elderly user's mental state and take appropriate measures.

[0072] The preferred embodiments have been shown and described, but should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A psychological assessment system based on feature profiling fusion using multimodal sensors, characterized in that: It includes an interaction module, a multimodal data acquisition module, a data processing module, and an evaluation result output module. The interactive module is used to generate questionnaires and record users' answers. The multimodal data acquisition module is used to collect relevant psychological activity data and sleep indicators of users during the question-answering process; The data processing module is used to process and analyze the collected data and sleep indicators, construct user feature profiles, integrate user feature profiles with question and answer data, and analyze user psychological state. The assessment result output module is used to output the assessment results of the questionnaire and push risk warnings to the client.

2. The feature profile fusion psychological assessment system based on multimodal sensors according to claim 1, characterized in that: The multimodal data acquisition module includes a voice acquisition unit, an interactive acquisition unit, a physiological signal acquisition unit, and a sleep data acquisition unit; The voice acquisition unit collects the user's voice signals during the question-answering process via a microphone; the interaction acquisition unit is used to collect the user's interaction data during the question-answering process, including the reaction time, answer time, pause ratio, and number of interruptions for each questionnaire question. The physiological signal acquisition unit and the sleep data acquisition unit are integrated into the wearable device. The physiological signal acquisition unit is used to collect physiological signals during the user's answering process, including heart rate, respiration, electrocardiogram and blood oxygen. The sleep data acquisition unit is used to collect sleep indicators before the user's assessment, including sleep quality, sleep duration, and sleep stage information.

3. The feature profile fusion psychological assessment system based on multimodal sensors according to claim 2, characterized in that: The data processing module includes a data preprocessing unit, a time-series alignment unit, an individualized baseline construction unit, a feature extraction unit, a feature profile generation unit, a credibility calculation unit, and a verification unit. The preprocessing unit cleans and denoises the data acquired by the multimodal data acquisition module; The time alignment unit aligns the answering time period of each questionnaire question with the physiological signals and interaction data of the corresponding time period to establish a unified time axis, and constructs an advance baseline window, an answer window, and a post-question reply window for each question and answer data. The individualized baseline construction unit establishes an individualized baseline model by statistically analyzing the mean and standard deviation of each physiological characteristic based on the subject's historical resting state data and physiological data during casual conversation. The feature extraction unit standardizes the physiological features corresponding to each questionnaire item based on the mean and standard deviation of the physiological features. The feature profile generation unit deeply integrates standardized physiological features, interaction features, and questionnaire response features, and uses an LSTM-FCNN hybrid model to concatenate the feature vectors of the three single-dimensional profiles and mine temporal correlations to output the user's psychological feature profile. The credibility calculation unit evaluates the credibility of the question and answer data for each questionnaire question based on the interaction data and standardized physiological characteristics, corrects the credibility of each question and answer data, and obtains the total score of the user's psychological state and risk level in the session. The verification unit is used to verify question-and-answer data with low credibility. The verification unit will ask the user relevant questions again and combine them with the corresponding physiological signals, interaction data and sleep indicators to re-verify the credibility of the question-and-answer data.

4. The feature profile fusion psychological assessment system based on multimodal sensors according to claim 3, characterized in that: The standardized expression for the physiological characteristic is: ， in, z i,k For the first i Question No. k Standardized values ​​of physiological characteristics x i,k For the first i Question No. k A physiological characteristic, μ k For the first k Individualized baseline mean of each physiological characteristic σ k For the first k Individualized baseline standard deviation of each physiological characteristic It is a very small constant.

5. The feature profile fusion psychological assessment system based on multimodal sensors according to claim 4, characterized in that: The expression for the total score of the psychological state is: ， in, M The total score represents the psychological state during the conversation. For the first i The weight of the question For the first i The credibility score of the question. For credibility correction function, For the first i The original psychological score of the question.

6. The feature profile fusion psychological assessment system based on multimodal sensors according to claim 5, characterized in that: The data processing module is also equipped with a storage unit for storing the data collected by the multimodal data acquisition module.

7. The feature profile fusion psychological assessment system based on multimodal sensors according to claim 6, characterized in that: It also includes a data security module that encrypts and stores all collected data and sets access control, allowing only authorized personnel to access the data through a specific secure channel.

Images