Interactive multi-modal artificial intelligence psychological behavior assessment method and system based on drawing analysis

Abstract

The invention discloses an interactive multi-modal artificial intelligence psychological behavior assessment method and system based on drawing analysis. The method comprises the steps that multi-modal data are synchronously collected, static drawing data, dynamic behavior data and physiological signal data of a user in the drawing process are obtained, and time alignment is conducted on the multi-modal data; analyzing the collected multi-modal data, feeding back an analysis result to the user in a suggestion or question form, then executing the data collection stage again to grasp a reaction result of the user to the suggestion or question, analyzing the collected multi-modal data, and then executing a fusion analysis stage; performing fusion analysis on data of three dimensions of static drawing, dynamic behaviors and physiological signals to generate a multi-modal feature representing a comprehensive psychological state of the user; and according to the multi-modal features representing the comprehensive psychological state of the user, in combination with psychological knowledge and a database, generating and outputting relative reports among individuals and psychological portraits in the individuals after multiple times of evaluation of the user by utilizing a natural language generation model. Starting from multiple dimensions, the defects of the current drawing psychological analysis technology can be effectively overcome, and the accuracy of psychological assessment is improved.

Description

Technical Field

[0001] This invention belongs to the field of computer artificial intelligence and psychological testing technology, and in particular, it is an interactive multimodal artificial intelligence psychological behavior assessment method and system based on drawing analysis. Background Technology

[0002] Current shortcomings in drawing analysis include: First, it requires a high level of professional experience. The test itself contains a wealth of information, necessitating the application of psychoanalytic theory to identify appropriate features to describe and interpret the drawing responses. This places high demands on the experience and skill of the administering personnel, thus limiting the test's versatility and the accuracy of its answers. Second, it is difficult to avoid subjective influence. Different administering personnel may yield inconsistent results for the same subject's drawings. Even experienced experts, without communicating with the subject, will inevitably interpret the results with a subjective bias. Third, the level of intelligence is insufficient. While some computer-aided drawing analysis systems exist, their feature analysis standards still rely on the administering personnel's clinical experience. They cannot automatically "learn" the features contained in drawings outside of specified samples and scenes, and these features cannot be generalized across different groups, resulting in relatively low standardization and failing to eliminate subjectivity and limitations.

[0003] To address the shortcomings of current painting-based psychological analysis and improve the accuracy of psychological assessments, patent application number 202510478646.9 discloses a House-Tree-Person (HTP) psychological assessment system and method based on artificial intelligence multimodal technology. By collecting static painting data and dynamic behavioral data and combining them with a psychological knowledge base, the system generates a HTP psychological assessment report, thereby improving the effectiveness of HTP assessments.

[0004] However, the above methods still have the following shortcomings: First, the data collection dimensions are lacking, and judging the user's psychological state based on the fluency and pauses of dynamic behavioral data is biased; second, the interactivity of the testing process is poor, and it is only based on some simple prompts to ensure the integrity of the drawing, and the single scenario makes it difficult to obtain the deep psychological state; third, in terms of result presentation, it is only based on the results of this test based on a horizontal comparison between individuals, and the presentation format is mainly text; fourth, the assessment medium is mainly the house-tree-person, but there are many mediums for drawing tests, including self-portraits, people in the rain, etc., and limiting oneself to one medium will limit the comprehensiveness of the information obtained. Summary of the Invention

[0005] The purpose of this invention is to address the problems existing in the prior art by providing an interactive multimodal artificial intelligence psychological behavior assessment method and system based on drawing analysis.

[0006] The technical solution for achieving the objective of this invention is: an interactive multimodal artificial intelligence psychological behavior assessment method based on drawing analysis, the method comprising:

[0007] Data collection phase: Multimodal data is collected synchronously to obtain static drawing data, dynamic behavior data and physiological signal data of users during the drawing process, and the multimodal data is time-aligned;

[0008] Analysis and feedback phase: Analyze the collected multimodal data and provide feedback to the user in the form of suggestions or questions. Then, perform the data collection phase again and analyze the collected multimodal data. Finally, perform the fusion analysis phase.

[0009] Fusion analysis phase: Fusion analysis is performed on data from three dimensions: static drawing, dynamic behavior, and physiological signals, to generate a multimodal feature that represents the user's comprehensive psychological state;

[0010] Report generation and output stage: Based on the multimodal characteristics that represent the user's comprehensive psychological state, combined with psychological knowledge and databases, natural language generation models are used to generate and output relative reports between individuals and intra-individual psychological profiles after multiple user assessments.

[0011] Furthermore, in the analysis feedback phase, the multimodal data collected during the analysis specifically includes:

[0012] For static painting data, a pre-trained convolutional neural network (CNN) model is used to extract deep features to obtain static visual features during the painting process;

[0013] For dynamic behavioral data, a pre-trained Long Short-Term Memory (LSTM) network model is used to process time-series data during the user's drawing process to extract dynamic behavioral features that reflect changes in psychological state.

[0014] For physiological signal data, the temporal and frequency domain features of the physiological signals are extracted. Using a facial region decomposition strategy and a pre-trained 3D convolutional model, the flow features are extracted to identify the dynamic changes of micro-expressions, and then the micro-expression tension index is calculated. The temporal and frequency domain features are spliced ​​and fused as physiological features.

[0015] Furthermore, the static painting data is the RGB image of key frames during the painting process; the dynamic behavior data is the time series of the painting process.

[0016] Furthermore, the pre-training of the convolutional neural network model specifically includes:

[0017] (1) Constructing a static painting dataset

[0018] Collect a number of static painting data to form an initial static painting dataset;

[0019] The initial static painting dataset is preprocessed to adapt to the input requirements of the convolutional neural network model, forming the final static painting dataset; the preprocessing includes at least size normalization;

[0020] (2) Construct an improved convolutional neural network model

[0021] ResNet50 is used, but the top fully connected layer is removed;

[0022] (3) The improved convolutional neural network model is trained using the final static painting dataset.

[0023] Furthermore, the pre-training of the Long Short-Term Memory network model specifically includes:

[0024] (1) Constructing a dynamic behavior dataset

[0025] Collect multiple sets of dynamic behavior data and dynamic behavior feature data pairs to form an initial dynamic behavior dataset;

[0026] The initial dynamic behavior dataset is preprocessed to adapt to the input requirements of the Long Short-Term Memory Network model, forming the final dynamic behavior dataset; the preprocessing includes at least missing value imputation, standardization and unification, and sequence alignment;

[0027] (2) Construct an improved long short-term memory network model

[0028] For the Long Short-Term Memory (LSTM) network model structure, add the following before the GRU recurrent neural network:

[0029] Temporal dependencies are captured in parallel using the Graph Attention Network (GAT) and the Temporal Convolutional Network (TCN).

[0030] The spatiotemporal features of each time point are obtained by fusing the features output by GAT and TCN through MLP;

[0031] (3) The improved long short-term memory network model is trained using the final dynamic behavior dataset.

[0032] Furthermore, the facial region decomposition strategy is used to generate an optical flow sequence, and the specific process includes:

[0033] Based on facial landmarks, the face is divided into at least six key regions, including eyebrows, eyes, nose, mouth, cheeks, and jaw:

[0034]

[0035] In the formula, Indicates the first A key area Represents the coordinates of facial landmarks;

[0036] Adjacent frames in micro-expression videos and Calculate the optical flow field of frame t. :

[0037]

[0038] in, The regularization coefficient is . These are the horizontal and vertical motion vectors, respectively, and T represents the total number of frames. This represents the gradient of the optical flow field;

[0039] Thus, the optical flow sequence is obtained. :

[0040]

[0041] In the formula, These are the width and height of the micro-expression video, respectively.

[0042] Furthermore, the input of the three-dimensional convolutional model is a facial region optical flow sequence, and the output is a stream light feature; wherein the facial region optical flow sequence is composed of optical flow sequences of 6 key regions spliced ​​together, and meets the input requirements of the three-dimensional convolutional model.

[0043] Furthermore, the calculation of the micro-expression tension index specifically includes:

[0044] physiological characteristics Streamer characteristics After normalization, the parts are spliced ​​together:

[0045]

[0046] In the formula, Indicates the features after splicing;

[0047] The micro-expression tension index, ranging from 0 to 1, is output through a fully connected layer and a sigmoid activation function.

[0048]

[0049] In the formula, Indicates the tension level in micro-expressions. This represents the Sigmoid activation function, used to map the output to the interval between 0 and 1. This represents the weight matrix of the fully connected layer, used to perform linear transformations on the fused features. This represents the bias term of the fully connected layer, used to adjust the baseline of the linear transformation.

[0050] Furthermore, in the fusion analysis phase, large-scale model inference combined with Monte Carlo training is performed on the three dimensions of static painting, dynamic behavior, and physiological signals to generate a multimodal feature representing the user's comprehensive psychological state, specifically including:

[0051] The data from the three dimensions are fed into a large model and trained in combination with a small model to filter out error data and duplicate data in the indicators, thus completing the data re-screening.

[0052] Monte Carlo tree search optimization was performed on the three-dimensional screening data and the data in the large model retrieval library. The optimization result with the highest score was selected as the optimal solution for dynamic multimodal fusion analysis.

[0053] Furthermore, the evaluation function for the score is:

[0054]

[0055] In the formula, This represents the accuracy of multi-hop inference in the TeCarlo simulation. This indicates the rating value.

[0056] Compared with the prior art, the significant advantages of this invention are:

[0057] (1) In terms of objectivity and consistency of assessment

[0058] This invention effectively eliminates subjective bias and achieves high consistency in results: Traditional painting tests rely heavily on the analyst's experience and theoretical orientation, and different experts may interpret the same painting significantly differently. This invention uses artificial intelligence models (CNN, LSTM) to quantify and calculate the static visual features, dynamic behavioral features, and physiological signal features of paintings, eliminating subjective assumptions and making the analysis process standardized and repeatable. Multiple assessments of the same sample yield consistent results, solving the industry problem of inconsistent results from different test administrators on the same painting, and greatly improving the credibility and reliability of the assessment results.

[0059] (2) In terms of evaluation efficiency and convenience

[0060] First, it significantly improves the efficiency of assessment and analysis: Traditionally, manually analyzing a drawing test typically requires experienced analysts to spend tens of minutes to several hours. This invention automates the entire process, from data collection to report generation, completing the entire process within minutes. This efficiency improvement is more than an order of magnitude, enabling the technology to be applied to rapid screening of large populations (such as new student enrollment or annual psychological surveys of corporate employees), solving the problems of "time-consuming and inefficient" traditional methods and saving a significant amount of professional human resources.

[0061] Secondly, it lowers the implementation threshold and operational complexity: the system integrates data collection, analysis, and report generation functions, significantly reducing the professional psychology background required of operators. Non-professionals can complete the test after simple training, solving the problem of "high professional experience requirements and limited test applicability" in traditional methods, which is conducive to the promotion and popularization of the technology.

[0062] (3) Effectiveness in assessing depth and accuracy

[0063] First, by using multimodal fusion, the dimensionality and accuracy of the assessment are significantly improved: Compared with existing technologies that only analyze static images (such as CNN models) and dynamic behavioral data of the painting process (such as pen pressure, speed, pauses, and erasures), this invention innovatively integrates physiological signal data (heart rate, facial expression, and blood pressure). This fusion analysis is equivalent to simultaneously interpreting the user's "creative result" and "creative process," and can evaluate the effectiveness of the assessment based on physiological signals, providing richer and more comprehensive psychological information.

[0064] Second, quantitative expectations: This invention is expected to improve the overall classification accuracy by 3-5 percentage points and significantly reduce the false negative rate (missed report rate), because certain psychological states are more evident in physiological and behavioral data than in static images.

[0065] Third, it enhances the ability to counter psychological defenses and improves the authenticity of the results; physiological signals (heart rate and facial expressions) are usually unconscious and difficult to conceal. By identifying these "defensive behavioral characteristics," the system can effectively determine whether the test subject has a tendency to conceal, and weight and calibrate the final assessment results, effectively mitigating the drawbacks of "obvious assessment objectives and results easily influenced by the test subject's subjective factors," thereby obtaining a more authentic and in-depth assessment of psychological state.

[0066] (4) Effects in terms of functionality and user experience

[0067] First, it enhances the fun and participation of the assessment: the digital painting experience is similar to a creative game, which is more likely to stimulate users' interest in participation than filling out lengthy and tedious questionnaires. It is especially suitable for teenagers and children, and solves the problem of traditional questionnaires being "lengthy and tedious, which greatly tests the patience of the test takers".

[0068] Secondly, the human-computer interaction feedback process helps to understand the user's deeper psychological and behavioral reactions.

[0069] Third, it generates structured and highly readable intelligent reports. The report generation module transforms cold data features into fluent, structured, and easy-to-understand text and dynamic graph visualization reports, covering psychological state overview, risk warnings, and personalized suggestions, which greatly improves the readability and practicality of the results and makes them easier for non-professionals (such as managers, parents, and other users) to understand.

[0070] (5) For static painting data, an improved convolutional neural network or its variant is used for modeling, leveraging its advantages in image recognition. This model is suitable for identifying and quantifying static visual features in paintings. For dynamic behavioral data, an improved long short-term memory network or its variant is used for modeling. This model excels at handling the time-series characteristics of the dynamic behavioral data stream and can effectively extract dynamic behavioral features reflecting changes in psychological state. For physiological signal data, a facial region decomposition strategy and three-dimensional convolution technique are used. This model can extract light flow features and effectively identify the user's emotional expression during the painting process. Thus, the accuracy of the final evaluation can be further improved.

[0071] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0072] Figure 1 This is a schematic diagram illustrating the principle of interactive multimodal artificial intelligence psychological behavior assessment based on drawing analysis in one embodiment.

[0073] Figure 2 This is a flowchart of an interactive multimodal AI-based psychological and behavioral assessment method based on drawing analysis, as shown in one embodiment. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0075] It should be noted that if the embodiments of the present invention involve descriptions such as "first" and "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0076] In one embodiment, combined Figure 1 and Figure 2 This paper provides an interactive, multimodal artificial intelligence-based psychological and behavioral assessment method based on drawing analysis, the method comprising:

[0077] Data collection phase: Multimodal data is collected synchronously to obtain static drawing data, dynamic behavior data and physiological signal data of users during the drawing process, and the multimodal data is time-aligned;

[0078] Here, data collection relies on hardware media including tablets, styluses, wristbands, cameras, etc. Any tool that can acquire static drawing data, dynamic behavioral data, and physiological signal data of the user during the drawing process is acceptable.

[0079] Analysis and feedback phase: Analyze the collected multimodal data and provide feedback to the user in the form of suggestions or questions. Then, perform the data collection phase again and analyze the collected multimodal data. Finally, perform the fusion analysis phase.

[0080] Fusion analysis phase: Fusion analysis is performed on data from three dimensions: static drawing, dynamic behavior, and physiological signals, to generate a multimodal feature that represents the user's comprehensive psychological state;

[0081] Report generation and output stage: Based on the multimodal characteristics that represent the user's comprehensive psychological state, combined with psychological knowledge and databases, natural language generation models are used to generate and output relative reports between individuals and intra-individual psychological profiles after multiple user assessments.

[0082] Furthermore, in one embodiment, the multimodal data collected during the analysis feedback phase specifically includes:

[0083] For static painting data, a pre-trained convolutional neural network model is used to extract deep features to obtain static visual features during the painting process.

[0084] For dynamic behavioral data, a pre-trained long short-term memory network model is used to process time-series data during the user's drawing process to extract dynamic behavioral features that reflect changes in psychological state.

[0085] For physiological signal data, the temporal and frequency domain features of the physiological signals are extracted. Using a facial region decomposition strategy and a pre-trained 3D convolutional model, the flow features are extracted to identify the dynamic changes of micro-expressions, and then the micro-expression tension index is calculated. The temporal and frequency domain features are spliced ​​and fused as physiological features.

[0086] Preferably, the static painting data is the RGB image of key frames during the painting process, including but not limited to: the compositional position, relative size, line thickness and continuity, and detail richness of houses, trees, and people.

[0087] Preferably, the dynamic behavior data is a time series of the painting process (including but not limited to pen pressure, pen speed, pause duration, coordinate trajectory, etc., denoted as time series). T is the time step. The action at time t can be the pressure applied to the pen, the speed of the stroke, the duration of the pause, or the coordinate trajectory, etc.

[0088] Preferably, the physiological signals include at least heart rate (HR), skin conductance (EDA), and respiratory rate (RR). For EDA, the time-domain features include at least the mean, standard deviation, and peak count, and the frequency-domain features include at least low-frequency (0.01-0.08Hz) energy, high-frequency (0.08-0.3Hz) energy, and high-low frequency ratio. For HR, the time-domain features include at least the mean heart rate and heart rate variability (SDNN), and the frequency-domain features include at least very low frequency (VLF, <0.04Hz), low-frequency (LF, 0.04-0.15Hz), and high-frequency (HF, 0.15-0.4Hz) energy and the LF / HF ratio. For RR, the time-domain features include at least the respiratory rate and the inspiratory / expiratory duration ratio.

[0089] Preferably, in some embodiments, the pre-training of the convolutional neural network model specifically includes:

[0090] (1) Constructing a static painting dataset

[0091] Collect a number of static painting data to form an initial static painting dataset;

[0092] The initial static painting dataset is preprocessed to adapt to the input requirements of the convolutional neural network model, forming the final static painting dataset; the preprocessing includes at least size normalization and filtering for noise reduction;

[0093] (2) Construct an improved convolutional neural network model

[0094] We use ResNet50 and remove the top fully connected layers, retaining only the convolutional + pooling backbone, including convolutional layers (output dimension 112×112×64), four residual blocks (56×56×256→7×7×2048), and a global average pooling layer (1×1×2048). The specific implementation is as follows:

[0095] Input preprocessed painting image To the ResNet50 backbone;

[0096] Hierarchical features are extracted using convolutional layers and residual blocks (lower layers: edges / textures, higher layers: composition / color combinations).

[0097] Global average pooling yields a 2048-dimensional deep feature vector;

[0098]

[0099] Where: GAP represents global average pooling. To remove the pre-trained ResNet50 from the top-level FC;

[0100] (3) The improved convolutional neural network model is trained using the final static painting dataset.

[0101] Preferably, in some embodiments, the pre-training of the Long Short-Term Memory network model specifically includes:

[0102] (1) Constructing a dynamic behavior dataset

[0103] Collect multiple sets of dynamic behavior data and dynamic behavior feature data pairs to form an initial dynamic behavior dataset;

[0104] The initial dynamic behavior dataset is preprocessed to adapt to the input requirements of the Long Short-Term Memory (LSTM) network model, forming the final dynamic behavior dataset. The preprocessing includes at least missing value imputation (using, but not limited to, linear interpolation) and standardization (e.g., ...). ( , (mean / standard deviation of the sequence) and sequence alignment (uniform time step T, padding with 0s for short sequences, truncation for long sequences);

[0105] (2) Construct an improved long short-term memory network model

[0106] For the Long Short-Term Memory (LSTM) network model structure, add the following before the GRU recurrent neural network:

[0107] Temporal dependencies are captured in parallel using Graph Attention Network (GAT, a neural network that processes graph-structured data, with the core function of assigning differentiated weights to node neighbors through a masked self-attention mechanism to efficiently learn spatial relationships in the graph) and Temporal Convolutional Network (TCN, a convolutional architecture oriented towards time series, with causal convolution + dilated convolution + residual connections as its core, capturing temporal dependencies in parallel and avoiding the gradient problem of RNNs).

[0108] The spatiotemporal features of each time point are obtained by fusing the features output by GAT and TCN through MLP;

[0109] Here, GRU, as a recurrent neural network (RNN) structure, can very effectively capture these long-term dependent features. GRU achieves the connection and modeling of temporal features through the following steps:

[0110] Input gate control: Each cell in the GRU controls the flow of information through input gates (reset gates) and update gates. The input gates determine how much new input information is retained at the current time, while the update gates control the degree to which information from past time steps is integrated with the current information.

[0111] Hidden state update: GRU uses hidden states to pass features from the previous time step to the current time step, and combines these with new features from the current time step to update the device state representation. In this way, the model can not only remember important historical information, but also make adaptive adjustments based on the current time series data.

[0112] Capturing long-term dependencies: GRU effectively avoids the vanishing gradient problem in traditional RNNs through its gating mechanism, enabling it to capture dependencies over long periods. In time-series data, the state of a device may be influenced by historical data over a long period, and GRU can learn and remember key historical information step by step.

[0113] (3) The improved long short-term memory network model is trained using the final dynamic behavior dataset.

[0114] Preferably, in some embodiments, the facial region decomposition strategy is used to generate an optical flow sequence, the specific process of which includes:

[0115] Based on facial landmarks (68 points), the face is divided into at least six key regions, including eyebrows, eyes, nose, mouth, cheeks, and jaw:

[0116]

[0117] In the formula, Indicates the first A key area Represents the coordinates of facial landmarks;

[0118] Adjacent frames in micro-expression videos and Calculate the optical flow field of frame t. :

[0119]

[0120] in, is the regularization coefficient (preferably 0.01). These are the horizontal and vertical motion vectors, respectively, and T represents the total number of frames. This represents the gradient of the optical flow field;

[0121] Thus, the optical flow sequence is obtained. :

[0122]

[0123] In the formula, These are the width and height of the micro-expression video, respectively.

[0124] Preferably, in some embodiments, the input of the three-dimensional convolutional model is a facial region optical flow sequence, and the output is a stream light feature; wherein the facial region optical flow sequence is composed of optical flow sequences of 6 key regions spliced ​​together, and meets the input requirements of the three-dimensional convolutional model.

[0125] Preferably, in some embodiments, calculating the micro-expression tension index specifically includes:

[0126] physiological characteristics Streamer characteristics After normalization, the parts are spliced ​​together:

[0127]

[0128] In the formula, Indicates the features after splicing;

[0129] The micro-expression tension index is output in the range of 0-1 through a fully connected layer and Sigmoid activation.

[0130]

[0131] In the formula, Indicates the tension level in micro-expressions. This refers to the Sigmoid function, which maps the output to the range of 0 to 1. This represents the weight matrix of the fully connected layer, used to perform linear transformations on the fused features. This represents the bias term of the fully connected layer, used to adjust the baseline of the linear transformation.

[0132] Preferably, in some embodiments, during the fusion analysis phase, large-scale model inference combined with Monte Carlo training is performed on the three dimensions of static painting, dynamic behavior, and physiological signals to generate a multimodal feature representing the user's comprehensive psychological state, specifically including:

[0133] The data from the three dimensions are fed into a large model and trained in combination with a small model to filter out error data and duplicate data in the indicators, thus completing the data re-screening.

[0134] Monte Carlo tree search optimization was performed on the three-dimensional screening data and the data in the large model retrieval library. The optimization result with the highest score was selected as the optimal solution for dynamic multimodal fusion analysis.

[0135] More preferably, in some embodiments, the evaluation function of the score is:

[0136]

[0137] In the formula, This represents the accuracy of multi-hop inference in the TeCarlo simulation. This represents the score. The weights here can also be adaptively adjusted based on other practical needs.

[0138] Preferably, in some embodiments, the content fed back to the user during the analysis and feedback phase includes:

[0139] Based on the completeness of the painting content, provide feedback on the painting elements and content (e.g., if a missing element is detected, provide feedback on whether it should be supplemented or improved; if a static image of a heavily painted roof is provided, provide feedback that there may be a certain degree of stress; provide feedback on physiological data showing that a surge in heart rate and facial tension occurred while painting a certain element).

[0140] Based on the combination of features identified by the pre-trained model that may indicate concealment behavior (e.g., an unusually slow drawing speed (deliberate control) accompanied by a persistently high heart rate and tense facial expressions (physiological reactions that are difficult to conceal), this "inconsistency" is marked as high concealment risk and the confidence level of the final evaluation result is downweighted).

[0141] In one embodiment, an interactive multimodal AI-based psychological and behavioral assessment system based on drawing analysis is provided, the system comprising:

[0142] The first module is used for synchronous acquisition of multimodal data, acquiring static painting data, dynamic behavior data and physiological signal data of users during the painting process, and performing time alignment on the multimodal data;

[0143] The second module is used to analyze the collected multimodal data and provide feedback to the user in the form of suggestions or questions. After that, the data collection phase is executed again, and the collected multimodal data is analyzed. Then, the fusion analysis phase is executed.

[0144] The third module is used to fuse and analyze data from three dimensions: static painting, dynamic behavior, and physiological signals, to generate a multimodal feature that represents the user's overall psychological state.

[0145] The fourth module is used to generate and output relative reports between individuals and intra-individual psychological profiles after multiple user assessments, based on multimodal characteristics that represent the user's comprehensive psychological state, combined with psychological knowledge and databases, using a natural language generation model.

[0146] Specific limitations regarding the interactive multimodal AI-based psychological and behavioral assessment system based on drawing analysis can be found in the limitations of the method for interactive multimodal AI-based psychological and behavioral assessment based on drawing analysis described above, and will not be repeated here. Each module in the aforementioned interactive multimodal AI-based psychological and behavioral assessment system based on drawing analysis can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0147] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements:

[0148] Data collection phase: Multimodal data is collected synchronously to obtain static drawing data, dynamic behavior data and physiological signal data of users during the drawing process, and the multimodal data is time-aligned;

[0149] Analysis and feedback phase: Analyze the collected multimodal data and provide feedback to the user in the form of suggestions or questions. Then, perform the data collection phase again and analyze the collected multimodal data. Finally, perform the fusion analysis phase.

[0150] Fusion analysis phase: Fusion analysis is performed on data from three dimensions: static drawing, dynamic behavior, and physiological signals, to generate a multimodal feature that represents the user's comprehensive psychological state;

[0151] Report generation and output stage: Based on the multimodal characteristics that represent the user's comprehensive psychological state, combined with psychological knowledge and databases, natural language generation models are used to generate and output relative reports between individuals and intra-individual psychological profiles after multiple user assessments.

[0152] For specific limitations on each stage, please refer to the limitations of the interactive multimodal AI psychological and behavioral assessment method based on drawing analysis mentioned above, which will not be repeated here.

[0153] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being implemented when executed by a processor:

[0154] Data collection phase: Multimodal data is collected synchronously to obtain static drawing data, dynamic behavior data and physiological signal data of users during the drawing process, and the multimodal data is time-aligned;

[0155] Analysis and feedback phase: Analyze the collected multimodal data and provide feedback to the user in the form of suggestions or questions. Then, perform the data collection phase again and analyze the collected multimodal data. Finally, perform the fusion analysis phase.

[0156] Fusion analysis phase: Fusion analysis is performed on data from three dimensions: static drawing, dynamic behavior, and physiological signals, to generate a multimodal feature that represents the user's comprehensive psychological state;

[0157] Report generation and output stage: Based on the multimodal characteristics that represent the user's comprehensive psychological state, combined with psychological knowledge and databases, natural language generation models are used to generate and output relative reports between individuals and intra-individual psychological profiles after multiple user assessments.

[0158] For specific limitations on each stage, please refer to the limitations of the interactive multimodal AI psychological and behavioral assessment method based on drawing analysis mentioned above, which will not be repeated here.

[0159] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention without departing from its spirit and scope should be included within the protection scope of the present invention.

Claims

1. An interactive, multimodal AI-based psychological and behavioral assessment method based on drawing analysis, characterized in that, The method includes: Data collection phase: Multimodal data is collected synchronously to obtain static drawing data, dynamic behavior data and physiological signal data of users during the drawing process, and the multimodal data is time-aligned; Analysis and feedback phase: Analyze the collected multimodal data and provide feedback to the user in the form of suggestions or questions. Then, perform the data collection phase again and analyze the collected multimodal data. Finally, perform the fusion analysis phase. Fusion analysis phase: Fusion analysis is performed on data from three dimensions: static drawing, dynamic behavior, and physiological signals, to generate a multimodal feature that represents the user's comprehensive psychological state; Report generation and output stage: Based on the multimodal characteristics that represent the user's comprehensive psychological state, combined with psychological knowledge and databases, natural language generation models are used to generate and output relative reports between individuals and intra-individual psychological profiles after multiple user assessments.

2. The interactive multimodal AI-based psychological and behavioral assessment method based on drawing analysis according to claim 1, characterized in that, In the analysis feedback phase, the multimodal data collected during the analysis specifically includes: For static painting data, a pre-trained convolutional neural network model is used to extract deep features to obtain static visual features during the painting process. For dynamic behavioral data, a pre-trained long short-term memory network model is used to process time-series data during the user's drawing process to extract dynamic behavioral features that reflect changes in psychological state. For physiological signal data, the temporal and frequency domain features of the physiological signals are extracted. Using a facial region decomposition strategy and a pre-trained 3D convolutional model, the flow features are extracted to identify the dynamic changes of micro-expressions, and then the micro-expression tension index is calculated. The temporal and frequency domain features are spliced ​​and fused as physiological features.

3. The interactive multimodal AI-based psychological and behavioral assessment method based on drawing analysis according to claim 1, characterized in that, The static painting data is the RGB image of key frames during the painting process; the dynamic behavior data is the time series of the painting process.

4. The interactive multimodal artificial intelligence psychological behavior assessment method based on drawing analysis according to claim 2, characterized in that, The pre-training of the convolutional neural network model specifically includes: (1) Constructing a static painting dataset Collect a number of static painting data to form an initial static painting dataset; The initial static painting dataset is preprocessed to adapt to the input requirements of the convolutional neural network model, forming the final static painting dataset; the preprocessing includes at least size normalization; (2) Construct an improved convolutional neural network model ResNet50 is used, but the top fully connected layer is removed; (3) The improved convolutional neural network model is trained using the final static painting dataset.

5. The interactive multimodal AI-based psychological and behavioral assessment method based on drawing analysis according to claim 2, characterized in that, The pre-training of the Long Short-Term Memory network model specifically includes: (1) Constructing a dynamic behavior dataset Collect multiple sets of dynamic behavior data and dynamic behavior feature data pairs to form an initial dynamic behavior dataset; The initial dynamic behavior dataset is preprocessed to adapt to the input requirements of the Long Short-Term Memory Network model, forming the final dynamic behavior dataset; the preprocessing includes at least missing value imputation, standardization and unification, and sequence alignment; (2) Construct an improved long short-term memory network model For the Long Short-Term Memory (LSTM) network model structure, add the following before the GRU recurrent neural network: Temporal dependencies are captured in parallel using the Graph Attention Network (GAT) and the Temporal Convolutional Network (TCN). The spatiotemporal features of each time point are obtained by fusing the features output by GAT and TCN through MLP; (3) The improved long short-term memory network model is trained using the final dynamic behavior dataset.

6. The interactive multimodal artificial intelligence psychological behavior assessment method based on drawing analysis according to claim 2, characterized in that, The facial region decomposition strategy is used to generate optical flow sequences, and the specific process includes: Based on facial landmarks, the face is divided into at least six key regions, including eyebrows, eyes, nose, mouth, cheeks, and jaw: In the formula, Indicates the first A key area Represents the coordinates of facial landmarks; Adjacent frames in micro-expression videos and Calculate the optical flow field of frame t. : in, The regularization coefficient is . These are the horizontal and vertical motion vectors, respectively, and T represents the total number of frames. This represents the gradient of the optical flow field; Thus, the optical flow sequence is obtained. : In the formula, These are the width and height of the micro-expression video, respectively.

7. The interactive multimodal AI-based psychological and behavioral assessment method based on drawing analysis according to claim 6, characterized in that, The input to the three-dimensional convolutional model is a facial region optical flow sequence, and the output is a stream light feature; wherein the facial region optical flow sequence is composed of optical flow sequences of 6 key regions spliced ​​together, and meets the input requirements of the three-dimensional convolutional model.

8. The interactive multimodal artificial intelligence psychological behavior assessment method based on drawing analysis according to claim 7, characterized in that, The calculation of the micro-expression tension index specifically includes: physiological characteristics Streamer characteristics After normalization, the parts are spliced ​​together: In the formula, Indicates the features after splicing; The micro-expression tension index, ranging from 0 to 1, is output through a fully connected layer and a sigmoid activation function. In the formula, Indicates the tension level in micro-expressions. This represents the Sigmoid activation function, used to map the output to the interval between 0 and 1. This represents the weight matrix of the fully connected layer, used to perform linear transformations on the fused features. This represents the bias term of the fully connected layer, used to adjust the baseline of the linear transformation.

9. The interactive multimodal artificial intelligence psychological behavior assessment method based on drawing analysis according to claim 2, characterized in that, In the fusion analysis phase, large-scale model inference combined with Monte Carlo training is performed across three dimensions: static drawing, dynamic behavior, and physiological signals, to generate a multimodal feature representing the user's comprehensive psychological state. Specifically, this includes: The data from the three dimensions are fed into a large model and trained in combination with a small model to filter out error data and duplicate data in the indicators, thus completing the data re-screening. Monte Carlo tree search optimization was performed on the three-dimensional screening data and the data in the large model retrieval library. The optimization result with the highest score was selected as the optimal solution for dynamic multimodal fusion analysis.

10. The interactive multimodal artificial intelligence psychological behavior assessment method based on drawing analysis according to claim 9, characterized in that, The evaluation function for the score is: In the formula, This represents the accuracy of multi-hop inference in the TeCarlo simulation. This indicates the rating value.

Images