Multi-modal psychological and behavioral analysis method and system
By performing time alignment, quality assessment, and dynamic weighted fusion on multimodal data, the problems of dynamic feature capture and modality fusion in multimodal psychological and behavioral analysis are solved, achieving high-precision and transparent multi-task analysis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-08
- Publication Date
- 2026-03-17
AI Technical Summary
Existing multimodal psychological and behavioral analysis technologies suffer from problems such as lack of dynamic feature capture, rigid modality fusion mechanisms, insufficient multi-task modeling capabilities, and lack of interpretability and compliance.
By establishing a unified time axis to align the time of multimodal data, calculating the quality vector at each time step, performing feature extraction and temporal coding, dynamically weighting and fusing the data, multi-task learning is achieved, and interpretable analysis results are output.
It achieves strong dynamic psychological modeling capabilities, low misjudgment rate, and has the ability to optimize modality fusion mechanism and transparent auditing, making it suitable for multi-dimensional analysis and applications in highly sensitive scenarios.
Smart Images
Figure CN121667697A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence analysis technology, specifically to a multimodal psychological and behavioral analysis method and system. Background Technology
[0002] In recent years, with the rapid development of artificial intelligence (AI) technology, its application areas have become increasingly widespread. Among them, applying AI technology to multimodal psychological and behavioral analysis has become a core research direction in the field of intelligent decision-making. It has been widely used in key scenarios such as job recruitment, education and training, and psychological counseling, and has achieved good application results.
[0003] In current technologies, psychological and behavioral analysis has gradually shifted from traditional scale assessments to intelligent analysis paradigms based on multimodal data. Some researchers have attempted to integrate single or multiple modal data, such as text, voice, and facial expressions, and use deep learning models to achieve trait recognition. Examples include extracting facial features through facial key point detection, analyzing emotional states based on voice signals, or interpreting behavioral tendencies by combining text semantics. However, research has found that existing multimodal psychological and behavioral analysis technologies still fall short of meeting the refined and highly reliable requirements of practical applications, and they suffer from the following shortcomings: (1) Lack of dynamic feature capture: Existing models mostly focus on frame-level static feature extraction, such as facial expressions and voice tone at a single moment, ignoring dynamic information in the time series dimension. Dynamic features such as the duration of micro-expressions, the changing trend of voice rhythm, and the temporal correlation of body movements are crucial for accurately identifying key traits such as impression management behavior and emotional fluctuations. However, existing technologies have failed to effectively model such temporal dependencies, resulting in limited analysis accuracy.
[0004] (2) Rigid modal fusion mechanism: Different modal data are easily affected by environmental interference during the acquisition process. For example, changes in lighting affect the quality of facial data and noise interferes with the integrity of speech signals, resulting in differences in the reliability of each modal data. However, existing multimodal fusion methods mostly adopt fixed weight allocation strategies and lack real-time evaluation and adaptive adjustment mechanisms for data quality. They cannot dynamically enhance the contribution of high-reliability modalities and suppress the interference of low-quality modalities, thus affecting the effectiveness of fusion features.
[0005] (3) Insufficient multi-task modeling capability: There are complex relationships between psychological and behavioral traits, such as the mutual influence between leadership and communication style, and growth mindset and emotional stability. In practical application scenarios, it is often necessary to assess multiple related traits at the same time. Existing technologies are mostly designed for single tasks and output single classification results, which makes it difficult to achieve multi-task collaborative modeling and cannot meet the multi-dimensional analysis needs of scenarios such as job interviews and educational assessments, thus limiting their applicability.
[0006] (4) Lack of Explainability and Compliance: AI interviews, educational assessments and other scenarios have extremely high requirements for the transparency and compliance of the technology, and it is necessary to clearly explain the generation logic of the analysis results to users and regulators. Most existing models are black box architectures, lacking explainable artificial intelligence (XAI) design, making it impossible to trace the key basis for trait assessment, and failing to establish a sound compliance audit mechanism. This makes it difficult to meet the regulatory requirements for data security and assessment fairness, thus limiting the large-scale application of the technology in highly sensitive scenarios. Summary of the Invention
[0007] The technical problem to be solved by this invention is that it discloses a multimodal psychological and behavioral analysis method and system to solve the problems of lack of dynamic feature capture, rigid modality fusion mechanism, insufficient multi-task modeling capability, and lack of interpretability and compliance in existing conventional multimodal analysis techniques.
[0008] To address the aforementioned technical problems, one objective of this invention is to disclose a multimodal psychological and behavioral analysis method, comprising the following steps: S1: Use multimodal data; S2: Establish a unified time axis to perform time alignment processing on the multimodal data, and calculate the quality vector of the time-aligned multimodal data at each time step; S3: Perform feature extraction on the time-aligned multimodal data at each time step to obtain the initial feature vector of the multimodal data; S4: Introduce the quality vector to perform time-series encoding on the initial feature vectors of each mode to obtain the feature vectors of each mode; S5: Calculate the dynamic weights of each mode based on the quality vector, and dynamically weight and fuse the feature vectors of each mode according to the dynamic weights to obtain the temporal fusion feature vector; S6: Perform pooling on the time-series fusion feature vector to form a global feature vector of fixed length; S7: Perform multi-task learning based on the global feature vector to achieve qualitative analysis of the psychological and behavioral characteristics of the target object; S8: Output feature importance analysis results and compliance audit records, and generate a readable and interpretable report.
[0009] To address the significant shortcomings of existing multimodal psychological and behavioral analysis technologies in areas such as dynamic feature modeling, adaptive modality fusion, multi-task collaboration, and compliance interpretability, this invention develops a multimodal psychological and behavioral analysis method that overcomes these technical bottlenecks. Based on temporal correlation, it achieves strong dynamic psychological modeling capabilities, possesses a superior modality fusion mechanism, and its model exhibits transparency and audit traceability. This method holds significant practical importance and application value.
[0010] Furthermore, in the multimodal psychological and behavioral analysis method of the present invention, in step S1, the multimodal data includes: video stream data, audio stream data, and text data; wherein, the video stream information includes facial video data; the audio stream data is acquired through a microphone, and the text data is acquired through ASR transcription or manual input.
[0011] Furthermore, in the multimodal psychological and behavioral analysis method of the present invention, in step S2, the time-aligned multimodal data is sliced according to a preset time window, and the quality vector of the time-aligned multimodal data is calculated moment by moment with one preset time window corresponding to one moment.
[0012] Furthermore, in the multimodal psychological and behavioral analysis method described in this invention, in step S2, for video stream data in multimodal data, the calculation dimensions of the quality vector include: lighting conditions, image clarity, occlusion degree, face visibility ratio, and action unit detection confidence; for audio stream data in multimodal data, the calculation dimensions of the quality vector include: signal-to-noise ratio, speech activity detection probability, silence ratio, and signal jitter rationality; for text data in multimodal data, the calculation dimensions of the quality vector include: ASR transcription confidence, word error rate, and unclear word ratio.
[0013] Furthermore, in the multimodal psychological and behavioral analysis method described in this invention, step S3 specifically includes: Feature extraction of video stream data: Based on facial key points and action units, extract micro-expression intensity, kinetic energy, duration, head pose trajectory and gaze trajectory, calculate statistical features and first-order and second-order dynamic features to obtain the initial feature vector f_face(t) of the video stream data. Feature extraction of audio stream data: extract fundamental frequency, formants, root mean square energy, harmonic noise ratio, frequency jitter, amplitude jitter, chroma features, Mel spectrum, spectral contrast, and tone network features, and calculate statistical and dynamic features to obtain the initial feature vector f_audio(t) of the audio stream data. Feature extraction of text data: A sentence encoding model is used to generate semantic vectors, and sentiment polarity, cognitive vocabulary, uncertainty vocabulary, and growth vocabulary indicators are extracted to obtain the initial feature vector f_text(t) of the text data.
[0014] Furthermore, in the multimodal psychological and behavioral analysis method of the present invention, in step S4, the temporal encoding adopts an LSTM model or a Transformer model, which uses the quality vector of each modality as the input parameter of the temporal encoding model to introduce the quality vector into the attention mechanism or gating unit, thereby performing temporal encoding processing on the initial feature vector of each modality to obtain the feature vector of each modality.
[0015] Furthermore, in the multimodal psychological and behavioral analysis method described in this invention, in step S5, the dynamic weights of each modality are calculated based on the mass vector using a softmax function or a lightweight gating network.
[0016] Furthermore, in the multimodal psychological and behavioral analysis method described in this invention, in step S5, the weighted fusion formula is:
[0017] Where e_fused(t) is the fused feature vector at time t, e m (t) is the eigenvector of the m-th mode at time t.
[0018] Furthermore, in the multimodal psychological and behavioral analysis method of the present invention, in step S7, the multi-task learning includes: a regression task and a classification task. The regression task is used to output the target object's interaction strategy strength, growth mindset index, self-confidence, and emotional stability. The classification task is used to output the target object's communication style and risk warning level.
[0019] Accordingly, another objective of this invention is to disclose a multimodal psychological and behavioral analysis system, which leverages this system to achieve strong dynamic psychological modeling capabilities, a superior modality fusion mechanism, and model transparency and audit traceability, thereby meeting the usage requirements of different application scenarios. This multimodal psychological and behavioral analysis system includes: The data acquisition module is used to acquire multimodal data; The synchronization and quality assessment module is used to establish a unified time axis, perform time alignment processing on multimodal data, and calculate the quality vector of multimodal data at each time step. The feature extraction module is used to extract features from the aligned multimodal data time-by-time to obtain the initial feature vector of the multimodal data; The temporal coding module is used to introduce the quality vector and perform temporal coding processing on the initial feature vector of each mode to obtain the feature vector of each mode; The dynamic fusion module is used to calculate the dynamic weights of each mode based on the quality vector, and to dynamically weight and fuse the feature vectors of each mode according to the dynamic weights to obtain the temporal fusion feature vector. The vector aggregation module is used to perform pooling processing on the temporal fusion feature vector to form a global feature vector of fixed length. The decision-making module performs multi-task learning based on the global feature vector to achieve qualitative analysis of the psychological and behavioral characteristics of the target object. The Explanation and Audit module is used to output feature importance analysis results and compliance audit records, generating readable explanation reports.
[0020] The beneficial effects of this invention are as follows: This invention proposes a novel multimodal psychological and behavioral analysis method, which solves the problems of lack of dynamic feature capture, rigid modality fusion mechanism, insufficient multi-task modeling capability, and lack of interpretability and compliance in existing conventional multimodal analysis techniques. Moreover, this multimodal psychological and behavioral analysis method has the advantages of strong dynamic psychological modeling capability, low misjudgment rate, stronger insight and analytical power, and better deployment flexibility. It has the characteristics of model transparency and audit traceability, which can meet the evaluation transparency requirements in different application scenarios. It has good prospects for promotion and application value.
[0021] Accordingly, the multimodal psychological and behavioral analysis system designed in this invention can be used to implement the multimodal psychological and behavioral analysis method described above, and it also has the aforementioned advantages and beneficial effects. Attached Figure Description
[0022] Figure 1 This is a flowchart illustrating the steps of one implementation of the multimodal psychological and behavioral analysis method described in this invention. Detailed Implementation
[0023] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.
[0024] To address the shortcomings of existing conventional multimodal analysis techniques, such as the lack of dynamic feature capture, rigid modality fusion mechanisms, insufficient multi-task modeling capabilities, and lack of interpretability and compliance, this invention designs a novel multimodal psychological and behavioral analysis method. This method boasts advantages such as strong dynamic psychological modeling capabilities, low false positive rate, enhanced insight and analytical power, and greater deployment flexibility. It also features model transparency and audit traceability, thereby meeting the evaluation transparency requirements in different application scenarios.
[0025] Please refer to Figure 1 As shown, this invention designs a novel multimodal psychological and behavioral analysis method, which includes the following steps: S1: Use multimodal data; S2: Establish a unified time axis to perform time alignment processing on the multimodal data, and calculate the quality vector of the time-aligned multimodal data at each time step; S3: Perform feature extraction on the time-aligned multimodal data at each time step to obtain the initial feature vector of the multimodal data; S4: Introduce the quality vector to perform time-series encoding on the initial feature vectors of each mode to obtain the feature vectors of each mode; S5: Calculate the dynamic weights of each mode based on the quality vector, and dynamically weight and fuse the feature vectors of each mode according to the dynamic weights to obtain the temporal fusion feature vector; S6: Perform pooling on the time-series fusion feature vector to form a global feature vector of fixed length; S7: Perform multi-task learning based on the global feature vector to achieve qualitative analysis of the psychological and behavioral characteristics of the target object; S8: Output feature importance analysis results and compliance audit records, and generate a readable and interpretable report.
[0026] In the multimodal psychological and behavioral analysis method designed in this invention, the multimodal data collected in step S1 above will be inconsistent depending on the specific application. Specifically, it needs to include data from at least two different modalities to ensure that the analysis results obtained by the multimodal psychological and behavioral analysis method have high accuracy.
[0027] It should be noted that, in some specific embodiments, in step S1 above, the multimodal data may specifically include: video stream data, audio stream data, and text data; wherein, the video stream information needs to include face video data; the audio stream data can be specifically obtained through microphone acquisition, and the text data is obtained through ASR (speech-to-text) transcription or manual input.
[0028] Accordingly, such as Figure 1 As shown, in actual setup, after obtaining the above multimodal data, it is also necessary to establish a unified time axis in step S2 above to perform time alignment processing on the multimodal data. Specifically, for the convenience of analysis, when completing the alignment, the audio data can be sliced at a frame rate of 10–40ms, the video data can be aligned at a frame rate of 25–60fps, and the text data can be aligned to the corresponding audio segment according to the ASR timestamp. After completing the above alignment processing, slicing processing can also be performed, for example, slicing the aligned multimodal data with a time window of 1–5s and an overlap rate of 25–50%.
[0029] In some specific embodiments, the data can also be sliced according to a preset time window, with one preset time window corresponding to one moment, and the quality vector of the time-aligned multimodal data can be calculated moment by moment.
[0030] It is important to note that when actually calculating the mass vectors of the aforementioned multimodal data, in order to obtain more accurate calculation results and ensure the validity of subsequent analysis results, different calculation dimensions need to be referenced for the mass vectors of different modalities. For example: When dealing with video stream data in multimodal data, the dimensions for calculating the quality vector can include: lighting conditions, image sharpness, degree of occlusion, proportion of visible faces, and confidence of action unit detection. When dealing with audio stream data in multimodal data, the dimensions for calculating the quality vector can include: signal-to-noise ratio, probability of speech activity detection, percentage of silence, and reasonableness of signal jitter. When dealing with text data in multimodal data, the dimensions for calculating the quality vector can include: ASR transcription confidence, word error rate, and percentage of unclear words.
[0031] It should be emphasized that the formulas for calculating vectors are for those skilled in the art to use when designing psychological and behavioral analyses. For example: As shown below, at time t, the quality vector of the video stream data is denoted as:
[0032] Among them, the above The above represents the illumination condition score at time t. The above represents the image sharpness score at time t. The score represents the degree of occlusion at time t; the above The above represents the facial visibility ratio score at time t; The confidence score for action unit detection at time t is represented.
[0033] In practical applications, the above lighting condition scoring Based on the average brightness of the face region in the current frame With target brightness The deviation between them can be measured and defined as:
[0034] The clip(·) function truncates the result to the range [0, 1].
[0035] Accordingly, the above image sharpness score Specifically, this can be achieved by calculating the variance of the Laplacian operator in the face region. As a sharpness indicator, and after normalization, the following is obtained:
[0036] in, The variance of the Laplacian operator for the face region at time t is calculated by performing a Laplacian convolution on the face region image and is used to measure the sharpness and clarity of the image frame. This represents the lowest acceptable resolution variance threshold in empirical data; This represents the highest detectable sharpness variance threshold in empirical data, used for linear normalization of the sharpness index so that the final sharpness score falls within the [0, 1] interval.
[0037] Accordingly, the above occlusion level score It can be obtained using the following formula:
[0038] in, Let be the area of the visible face region at time t. Let be the area of the occluded region at time t.
[0039] Accordingly, the above facial visibility ratio score It can be obtained using the following formula:
[0040] in, Let be the projected area of the face in the image at time t. Let be the area of the entire frame at time t.
[0041] Accordingly, the confidence score of the above action unit was detected. The following formula is used to obtain it:
[0042] in, The confidence score is the score of the action unit detected in the current frame. The confidence score of the action unit detection is obtained by averaging the confidence scores of the N action units detected in the current frame. Furthermore, as shown below, the quality vector of the audio stream data at time t is denoted as:
[0043] in, The signal-to-noise ratio score at time t is represented. This represents the probability of voice activity detection at time t; The score represents the percentage of silence at time t; The rating represents the reasonableness score of signal jitter at time t.
[0044] In practical applications, signal-to-noise ratio (SNR) scoring The specific method used is as follows: Assume the signal-to-noise ratio of the current audio segment is Within the scope of experience Normalization is performed internally:
[0045] In practical applications, the probability of voice activity detection The specific method used is as follows: Directly using the speech activity probability output by the VAD model :
[0046] In practical applications, the percentage of quiet operation is scored. The specific method used is as follows: Let the percentage of silent frames within the current time window be . ,but:
[0047] In practical applications, signal jitter rationality scoring The specific method used is as follows: The index is based on fundamental frequency jitter or amplitude jitter. Reference range Deviation measurement:
[0048] Furthermore, as described below, at time t, the quality vector of the text data is denoted as:
[0049] in, This represents the ASR transcription confidence score at time t; The word error rate score at time t; The score represents the percentage of unidentified words at time t.
[0050] In practical applications, ASR transcription confidence scoring The specific method used is as follows:
[0051] in, The confidence score is directly used from the speech recognition engine. In practical applications, the word error rate score is used. The specific method used is as follows:
[0052] in, Word error rate that can be obtained in a scenario where samples are manually proofread.
[0053] In practical applications, the percentage of unclear words in the scoring is important. The specific method used is as follows:
[0054] in, This represents the percentage of unknown or unidentified words within the current time window. In some embodiments, the scores of the above-mentioned dimensions can be weighted to obtain a scalar quality score for the modality. For example, the overall quality score of a video modality can be expressed as follows: :
[0055] The above This represents the weight coefficient of the k-th quality indicator in the video modality, used to reflect the relative importance of this indicator in the overall quality score; This represents the k-th quality score of the video modality at time t. It is obtained after normalization based on indicators such as lighting conditions, image clarity, degree of occlusion, proportion of visible faces, and confidence of action unit detection. The value range is limited to [0, 1].
[0056] In practical applications, weights The weighting can be determined based on empirical calibration, statistical learning, or data-driven methods; this invention does not impose any limitations on this. The above weighted calculation method applies to both audio and text modalities, i.e.: Overall quality score of audio modality It can be represented as:
[0057] in, Quality indicators include signal-to-noise ratio, probability of voice activity detection, percentage of silence, and reasonableness of signal jitter. Overall quality score of text modality It can be represented as
[0058] in, It includes quality indicators such as ASR transcription confidence, word error rate, and percentage of unclear words.
[0059] For audio and text modalities, the calculation process is the same as for video modalities, i.e., each is scored item by item based on its respective quality index, and then weighted and normalized to obtain the modality at time step [time]. The overall quality score is calculated. This invention does not limit the specific weight configuration or the number of indicators for each modality; it can be expanded or adjusted according to the needs of actual scenarios.
[0060] Accordingly, after calculating the quality vector of the multimodal data, it is also necessary to perform feature extraction on the time-aligned multimodal data time-by-time in the multimodal psychological and behavioral analysis method designed in this invention to obtain the initial feature vector of the multimodal data, which specifically includes the following: Feature extraction of video stream data: Based on facial key points and action units, extract micro-expression intensity, kinetic energy, duration, head pose trajectory and gaze trajectory, calculate statistical features and first-order and second-order dynamic features to obtain the initial feature vector f_face(t) of the video stream data. Feature extraction of audio stream data: extract fundamental frequency, formants, root mean square energy, harmonic noise ratio, frequency jitter, amplitude jitter, chroma features, Mel spectrum, spectral contrast, and tone network features, and calculate statistical and dynamic features to obtain the initial feature vector f_audio(t) of the audio stream data. Feature extraction of text data: A sentence encoding model is used to generate semantic vectors, and sentiment polarity, cognitive vocabulary, uncertainty vocabulary, and growth vocabulary indicators are extracted to obtain the initial feature vector f_text(t) of the text data.
[0061] Accordingly, in the multimodal psychological and behavioral analysis method designed in this invention, after obtaining the initial feature vector f_face(t) of video stream data, the initial feature vector f_audio(t) of audio stream data, and the initial feature vector f_text(t) of text data at different times t in step S3, in step S4, a quality vector is introduced to perform temporal encoding processing on the initial feature vectors of each modality to obtain the feature vectors of each modality.
[0062] In practical applications, the aforementioned temporal coding can specifically adopt an LSTM model or a Transformer model, which can use the quality vector of each modality as the input parameter of the temporal coding model to introduce the quality vector into the attention mechanism or gating unit, thereby performing temporal coding processing on the initial feature vector of each modality to obtain the feature vector of each modality.
[0063] Therefore, in step S5 of the multimodal psychological and behavioral analysis method designed in this invention, after calculating the dynamic weights of each modality based on the above-mentioned quality vectors, the feature vectors of each modality obtained in step S4 are dynamically weighted and fused according to the calculated dynamic weights to obtain the temporal fusion feature vector.
[0064] It should be noted that in practical applications, the dynamic weights of each mode can be calculated based on the mass vector using a softmax function or a lightweight gating network; specifically, the formula for the softmax function is set as follows:
[0065] Among them, w m (t) represents the weight of the m-th mode at time t, q m (t) is the mass vector of the m-th mode at time t, γ is the weighting coefficient, and b is the bias term.
[0066] Accordingly, in step S5, the above dynamic weighted fusion formula is:
[0067] Where e_fused(t) is the fused feature vector at time t, e m (t) is the eigenvector of the m-th mode at time t.
[0068] It should be noted that during the dynamic weighted fusion process described above, weight renormalization and imputation can also be performed on the missing modalities. The imputation method for the missing modalities is forward filling or attention imputation.
[0069] Based on this, after obtaining the aforementioned temporal fusion feature vector, it is necessary to perform pooling processing on the temporal fusion feature vector in step S6 to form a global feature vector of fixed length. Specifically, the pooling processing may include statistical pooling and attention pooling to attach quality summary information when forming the global feature vector of fixed length. The quality summary information may specifically include the average reliability of each modality and the data missing rate.
[0070] In summary, based on the final obtained global feature vector, multi-task learning can be performed to complete the qualitative analysis of the psychological and behavioral characteristics of the target object. In order to ensure the accuracy of the final output analysis results, multi-task learning can be carried out through nonlinear decision-making in step S7. This nonlinear decision-making can be implemented using XGBoost or GBDT models. This multi-task learning can specifically include regression tasks and classification tasks.
[0071] The regression task is used to output the target object's interaction strategy strength, growth mindset index, self-confidence, and emotional stability; the classification task is used to output the target object's communication style and risk warning level.
[0072] Accordingly, in order to achieve the characteristics of model transparency and audit traceability, in step S8 of the multimodal psychological and behavioral analysis method designed in this invention, the feature importance analysis results and compliance audit records can be output, and a readable and interpretable report can be generated.
[0073] Accordingly, in order to facilitate the implementation of the multimodal psychological and behavioral analysis method described above, the present invention also discloses a multimodal psychological and behavioral analysis system, which is used to perform the multimodal psychological and behavioral analysis method described above.
[0074] In practical applications, this multimodal psychological and behavioral analysis system can specifically include: The data acquisition module is used to acquire multimodal data; The synchronization and quality assessment module is used to establish a unified time axis, perform time alignment processing on multimodal data, and calculate the quality vector of multimodal data at each time step. The feature extraction module is used to extract features from the aligned multimodal data time-by-time to obtain the feature vector of the multimodal data; The temporal coding module is used to introduce the quality vector and perform temporal coding processing on the feature vectors of each mode to obtain the feature vectors of each mode. The dynamic fusion module is used to calculate the dynamic weights of each mode based on the quality vector, and to dynamically weight and fuse the feature vectors of each mode according to the dynamic weights to obtain the temporal fusion feature vector. The vector aggregation module is used to perform pooling processing on the temporal fusion feature vector to form a global feature vector of fixed length. The decision-making module performs multi-task learning based on the global feature vector to achieve qualitative analysis of the psychological and behavioral characteristics of the target object. The Explanation and Audit module is used to output feature importance analysis results and compliance audit records, generating readable explanation reports.
[0075] It should be noted that, in some specific implementations, in order to facilitate information transmission, the multimodal psychological and behavioral analysis system designed in this invention may also include a service interface module, which can be used to provide interface services for data uploading, inference calculation, report acquisition, and audit management.
[0076] Accordingly, the multimodal psychological and behavioral analysis system supports two deployment modes: online real-time streaming processing and offline asynchronous batch processing. It can be deployed at the edge or in the cloud, and the collected sensitive information is minimized and anonymized. In addition, the multimodal psychological and behavioral analysis system may also include a front-end display module for displaying the quantitative scores of psychological and behavioral characteristics, interval confidence levels, playback of key data segments, and feature importance heatmaps.
[0077] Therefore, in view of the shortcomings of existing multimodal analysis technology, this invention proposes a new multimodal psychological and behavioral analysis method and system, which has the advantages of strong dynamic psychological modeling ability, low misjudgment rate, stronger insight and analytical ability, and better deployment flexibility. It also has the characteristics of model transparency and audit traceability to meet the evaluation transparency requirements in different application scenarios.
[0078] As can be seen from the above description, the multimodal psychological and behavioral analysis method and system designed in this invention have the following beneficial effects: 1. Strong dynamic psychological modeling capability: This multimodal psychological and behavioral analysis method and system can use LSTM or Transformer models to perform temporal encoding on the feature vectors of each modality to establish a time series, thereby capturing the temporal correlation of micro-expressions, speech prosody and semantic changes, realizing the evolution from "static feature recognition" to "dynamic behavioral understanding", with significant improvement in accuracy and consistency; 2. Adaptive weighting: This multimodal psychological and behavioral analysis method and system introduces a modal reliability assessment scheme, which can automatically adjust the weights according to factors such as illumination, noise, and ASR error rate, thereby ensuring the robustness of the analysis results in low-quality environments and reducing the misjudgment rate; 3. Multi-task psychological trait output: This multimodal psychological and behavioral analysis method and system can simultaneously output multi-dimensional psychological indicators such as impression management (IM) strategy distribution, growth mindset, self-confidence, and communication stability. Compared with traditional single classification models, it has higher behavioral insight and explanatory power. 4. Nonlinear decision-making and high fitting ability: This multimodal psychological and behavioral analysis method and system can adopt nonlinear decision-making, and its second stage can use XGBoost / GBDT for nonlinear regression, which can effectively capture the interaction effects between different modal features, and its results are better than those of deep models used alone.
[0079] 5. Model Transparency and Audit Traceability: This multimodal psychological and behavioral analysis method and system can clearly indicate the contribution of each modality and feature to psychological indicators and retain audit records, meeting the transparency requirements of AI interviews and educational assessments; 6. Scalability and Deployment Flexibility: This multimodal psychological and behavioral analysis method and system can support deployment at the edge and in the cloud, can perform real-time inference or asynchronous batch analysis, and can be extended to other psychological constructs, which has great potential for industrial applications.
[0080] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.
Claims
1. A multimodal psychological and behavioral analysis method, characterized in that, Including the following steps: S1: Use multimodal data; S2: Establish a unified time axis to perform time alignment processing on the multimodal data, and calculate the quality vector of the time-aligned multimodal data at each time step; S3: Perform feature extraction on the time-aligned multimodal data at each time step to obtain the initial feature vector of the multimodal data; S4: Introduce the quality vector to perform time-series encoding on the initial feature vectors of each mode to obtain the feature vectors of each mode; S5: Calculate the dynamic weights of each mode based on the quality vector, and dynamically weight and fuse the feature vectors of each mode according to the dynamic weights to obtain the temporal fusion feature vector; S6: Perform pooling on the time-series fusion feature vector to form a global feature vector of fixed length; S7: Perform multi-task learning based on the global feature vector to achieve qualitative analysis of the psychological and behavioral characteristics of the target object; S8: Output feature importance analysis results and compliance audit records, and generate a readable and interpretable report.
2. The multimodal psychological and behavioral analysis method according to claim 1, characterized in that, In step S1, the multimodal data includes: video stream data, audio stream data, and text data; wherein, the video stream information includes face video data; the audio stream data is obtained through microphone acquisition, and the text data is obtained through ASR transcription or manual input.
3. The multimodal psychological and behavioral analysis method according to claim 1, characterized in that, In step S2, the time-aligned multimodal data is sliced according to a preset time window, and the quality vector of the time-aligned multimodal data is calculated moment by moment, with one preset time window corresponding to one moment.
4. The multimodal psychological and behavioral analysis method according to claim 2, characterized in that, In step S2, for video stream data in multimodal data, the dimensions for calculating the quality vector include: lighting conditions, image clarity, degree of occlusion, proportion of visible faces, and confidence of action unit detection; for audio stream data in multimodal data, the dimensions for calculating the quality vector include: signal-to-noise ratio, probability of speech activity detection, proportion of silence, and reasonableness of signal jitter; for text data in multimodal data, the dimensions for calculating the quality vector include: ASR transcription confidence, word error rate, and proportion of unclear words.
5. The multimodal psychological and behavioral analysis method according to claim 2, characterized in that, In step S3, specifically: Feature extraction of video stream data: Based on facial key points and action units, extract micro-expression intensity, kinetic energy, duration, head pose trajectory and gaze trajectory, calculate statistical features and first-order and second-order dynamic features to obtain the initial feature vector f_face(t) of the video stream data. Feature extraction of audio stream data: extract fundamental frequency, formants, root mean square energy, harmonic noise ratio, frequency jitter, amplitude jitter, chroma features, Mel spectrum, spectral contrast, and tone network features, and calculate statistical and dynamic features to obtain the initial feature vector f_audio(t) of the audio stream data. Feature extraction of text data: A sentence encoding model is used to generate semantic vectors, and sentiment polarity, cognitive vocabulary, uncertainty vocabulary, and growth vocabulary indicators are extracted to obtain the initial feature vector f_text(t) of the text data.
6. The multimodal psychological and behavioral analysis method according to claim 5, characterized in that, In step S4, the temporal coding adopts an LSTM model or a Transformer model, which uses the quality vector of each modality as the input parameter of the temporal coding model to introduce the quality vector into the attention mechanism or gating unit, so as to perform temporal coding processing on the initial feature vector of each modality to obtain the feature vector of each modality.
7. The multimodal psychological and behavioral analysis method according to claim 1, characterized in that, In step S5, the dynamic weights of each mode are calculated based on the mass vector using a softmax function or a lightweight gating network.
8. The multimodal psychological and behavioral analysis method according to claim 1, characterized in that, In step S5, the weighted fusion formula is: Where e_fused(t) is the fused feature vector at time t, e m (t) is the eigenvector of the m-th mode at time t.
9. The multimodal psychological and behavioral analysis method according to claim 1, characterized in that, In step S7, the multi-task learning includes a regression task and a classification task. The regression task is used to output the target object's interaction strategy strength, growth mindset index, self-confidence, and emotional stability. The classification task is used to output the target object's communication style and risk warning level.
10. A multimodal psychological and behavioral analysis system, characterized in that, include: The data acquisition module is used to acquire multimodal data; The synchronization and quality assessment module is used to establish a unified time axis, perform time alignment processing on multimodal data, and calculate the quality vector of multimodal data at each time step. The feature extraction module is used to extract features from the aligned multimodal data time-by-time to obtain the initial feature vector of the multimodal data; The temporal coding module is used to introduce the quality vector and perform temporal coding processing on the initial feature vector of each mode to obtain the feature vector of each mode; The dynamic fusion module is used to calculate the dynamic weights of each mode based on the quality vector, and to dynamically weight and fuse the feature vectors of each mode according to the dynamic weights to obtain the temporal fusion feature vector. The vector aggregation module is used to perform pooling processing on the temporal fusion feature vector to form a global feature vector of fixed length. The decision-making module performs multi-task learning based on the global feature vector to achieve qualitative analysis of the psychological and behavioral characteristics of the target object. The Explanation and Audit module is used to output feature importance analysis results and compliance audit records, generating readable explanation reports.