A multi-modal psychological assessment system and method based on drawing and handwriting

Abstract

The application discloses a kind of multi-modal psychological evaluation system and method based on painting and handwriting.The method collects the multi-modal data of user when painting and writing operation is carried out on interactive digital drawing board, including painting static image, dynamic handwriting track and writing text content data;Image, dynamic and text features are extracted respectively, and cross-modal semantic conflict detection is carried out;If ideographic conflict is identified, orthogonal projection is used for semantic decoupling, and conflict feature matrix representing subconscious contradiction and consistent feature matrix representing conscious cooperation are extracted, and they are integrated into multi-modal features;Finally, the intention of user evaluation is obtained, and the intention and multi-modal features are input into psychological portrait model reasoning, and psychological evaluation report is generated accordingly.The scheme of the application can effectively overcome the feature masking problem caused by user defense psychology, accurately excavate the real emotion of subconscious, and significantly improve the objectivity and accuracy of psychological evaluation.

Description

Technical Field

[0001] This invention relates to the field of psychological assessment technology, specifically to a multimodal psychological assessment system and method based on drawing and handwriting. Background Technology

[0002] Psychological testing is a crucial foundation for mental health services, mental state screening, career planning, and self-exploration. Traditional psychological testing methods primarily rely on self-report scales (such as the SCL-90, MMPI, and 16PF), requiring users to read each written question and provide subjective answers.

[0003] For children, users with lower levels of education, those with speech difficulties, or those prone to social anxiety, it is difficult to accurately understand the meaning of the questions, leading to biased test results. At the same time, some users have strong defensive psychology and are easily driven by social approval when facing sensitive questions, deliberately concealing their true emotions and making unrealistic self-serving choices, resulting in a high false negative or false positive rate in the test results.

[0004] To reduce users' psychological defenses, projective psychological tests (such as the House-Tree-Person (HTP) and Dynamic Family Drawing) are widely used in clinical psychology. These tests guide users to draw freely, non-invasively projecting their subconscious emotions, conflicts, and personality traits onto the canvas. However, traditional projective tests rely heavily on the experience of human experts, resulting in high subjectivity, low analytical efficiency, and difficulty in standardization and generalization.

[0005] With the development of computer vision and pattern recognition technologies, some automated drawing psychology analysis systems have emerged in recent years. However, in actual psychological assessments, users with strong defensive tendencies often exhibit inconsistencies between their conscious (cognitive) and subconscious (instinctive projection) expressions. For example, under cognitive control, a user may write words expressing positive emotions (such as "I'm fine"), but when writing these words or drawing, the subconsciously driven dynamic handwriting trajectories, such as pen pressure, speed, or pauses, reveal extreme anxiety, repression, or a sense of powerlessness.

[0006] Existing multimodal fusion technologies typically employ simple feature splicing or fully connected weighted fusion. When faced with the aforementioned cross-modal semantic conflicts, the consistent features of conscious awareness and the contradictory features of subconscious awareness often cancel each other out or mask each other at the algorithm level, or are directly treated as noise signals and filtered out by the model. This results in existing systems being unable to effectively capture, separate, and identify the deep contradictions generated by users' psychological defense mechanisms, and are unable to truly uncover the users' hidden subconscious troubles, thus severely limiting the accuracy and objectivity of multimodal psychological assessments. Summary of the Invention

[0007] In view of the above-mentioned shortcomings mentioned in the background art, the purpose of this invention is to provide a multimodal psychological assessment method, system, computer device, and computer-readable storage medium based on drawing and handwriting.

[0008] This invention provides a multimodal psychological assessment method based on drawing and handwriting, the method comprising the following steps:

[0009] Collect multimodal data during the user's drawing and writing operations on the interactive digital drawing board, including static image data of drawing, dynamic handwriting trajectory data, and written text content data;

[0010] Image features, dynamic features, and text features are extracted from the static image data of the painting, the dynamic handwriting trajectory data, and the written text content data, respectively.

[0011] Cross-modal semantic conflict detection is performed on the image features, dynamic features and text features. When a semantic conflict is detected, orthogonal projection is used to decouple the semantics to extract the conflict feature matrix representing subconscious contradictions and the consistency feature matrix representing conscious coordination. The conflict feature matrix and the consistency feature matrix are then integrated into multimodal features.

[0012] The system obtains the user's assessment intent, inputs the assessment intent and the multimodal features into the psychological profile model, infers the multidimensional psychological state assessment results, and generates a psychological assessment report accordingly.

[0013] This invention also provides a multimodal psychological assessment system based on drawing and handwriting, the system comprising:

[0014] The data acquisition module is used to collect multimodal data during the user's drawing and writing operations on the interactive digital drawing board. The multimodal data includes static image data of the drawing, dynamic handwriting trajectory data, and written text content data.

[0015] The feature extraction module is used to extract image features, dynamic features, and text features from the static image data of the painting, the dynamic handwriting trajectory data, and the written text content data, respectively.

[0016] The conflict detection and decoupling module is used to perform cross-modal semantic conflict detection on the image features, the dynamic features and the text features. When a semantic conflict is detected, orthogonal projection is used to perform semantic decoupling to extract the conflict feature matrix representing subconscious contradictions and the consistency feature matrix representing conscious coordination. The conflict feature matrix and the consistency feature matrix are then integrated into multimodal features.

[0017] The reasoning report module is used to obtain the user's assessment intent, input the assessment intent and the multimodal features into the psychological profile model, reason to obtain the multidimensional psychological state assessment results, and generate a psychological assessment report accordingly.

[0018] The present invention also provides a computer device including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method as described in any of the preceding claims.

[0019] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method as described in any of the preceding claims.

[0020] This invention, through cross-modal semantic conflict detection and semantic decoupling based on orthogonal projection, effectively separates contradictory signals into a consistent feature matrix representing conscious coordination and a conflict feature matrix representing subconscious contradiction when semantic conflicts are identified. This process fully preserves the true psychological conflict concealed by linguistic disguise. Compared to existing technologies, this invention effectively overcomes the bias in psychological assessments caused by semantic inconsistency, significantly improving the objectivity and accuracy of assessments in situations involving strong defense mechanisms or verbal expression disorders. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a multimodal psychological assessment method based on drawing and handwriting disclosed in an embodiment of the present invention.

[0022] Figure 2 This is a schematic diagram of the structure of the psychological profiling model disclosed in an embodiment of the present invention;

[0023] Figure 3 This is a schematic diagram of the structure of a multimodal psychological assessment system based on drawing and handwriting disclosed in an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of the structure of the conflict detection and decoupling module 30 disclosed in an embodiment of the present invention;

[0025] Figure 5 This is a schematic diagram of the matrix construction unit 304 disclosed in an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0027] Reference Figure 1 As shown, this invention provides a multimodal psychological assessment method based on drawing and handwriting, the method comprising the following steps:

[0028] S1 collects multimodal data during the user's drawing and writing operations on the interactive digital drawing board, including static image data of the drawing, dynamic handwriting trajectory data, and written text content data;

[0029] This invention provides users with an interactive digital drawing board based on a high-precision electromagnetic induction panel or capacitive touchscreen. Users can freely draw and write on the blank canvas using their fingers or a stylus. Unlike traditional fixed-topic psychological assessments, this invention does not impose any preset restrictions on the drawing themes or writing content. Users can express themselves freely according to their own wishes, thereby reducing psychological defenses and inducing genuine subconscious projection.

[0030] During user operation, the data acquisition hardware captures physical signals in real time at a fixed sampling rate (e.g., 100Hz) and records the timing data of the entire operation process. Specifically, the acquired multimodal data includes:

[0031] Static image data for drawing: This refers to the pixel matrix that is finally displayed on the canvas after the user completes the drawing or writing operation. This static image retains visual information such as the spatial layout, color distribution, line direction, and overlapping relationships of each element in the picture.

[0032] Dynamic handwriting trajectory data: This refers to the spatiotemporal sequence of a user's behavior at every moment from the moment they put down the pen to the moment they lift it. For each sampling point, at least the following information should be recorded: time coordinates. This indicates the position of the pen tip on the drawing board; timestamp Used for sorting and speed calculation; pen tip pressure value Feedback is provided by a pressure sensor, with a preferred pressure level of 0-4096 levels. The pressure level reflects the user's emotional activation level while writing; brush color coding. The system records whether and when the user switches colors during the drawing process. This dynamic trajectory data, sorted by timestamp, forms a complete stroke sequence that can characterize the user's unconscious behavioral features, such as hesitation, acceleration, pauses, and sudden changes in stress.

[0033] Handwritten text data: When a user writes text (such as words or short phrases) on the drawing board, the writing dynamics are preserved through trajectory recording. Simultaneously, online handwriting recognition (HWR) technology or offline recognition models are used to extract character regions from the handwriting trajectory and convert them into Unicode text. This text content reflects the linguistic information the user is consciously willing to express, such as "I'm fine," "tired," or "confused," and can serve as a conscious benchmark for subsequent cross-modal semantic conflict detection.

[0034] It should be noted that the data collection was conducted synchronously, meaning that each sampling point was timestamped using the same clock reference. This ensures accurate alignment of the temporal relationships between static images, dynamic trajectories, and textual semantics during subsequent analysis. All collected raw data underwent preprocessing such as denoising and normalization before being temporarily stored in the system cache or cloud database.

[0035] S2, extract image features, dynamic features, and text features from the static image data of the painting, the dynamic handwriting trajectory data, and the written text content data, respectively;

[0036] Specifically as follows:

[0037] (1) Input the static image data of the painting (i.e., the pixel matrix of the final canvas) into a pre-trained computer vision model. The model can adopt a structure based on a deep convolutional neural network, such as a residual network (ResNet) or a vision transformer. The model performs end-to-end feature encoding on the image through multi-layer convolution or self-attention mechanism, and outputs a high-dimensional image feature vector.

[0038] During the extraction process, the model automatically identifies and encodes the following visual semantic information that can be used for psychological projection analysis:

[0039] Symbolic targets: such as whether the image contains typical projected objects such as houses, trees, people, the sun, and animals, and output the detection confidence level for each category;

[0040] Spatial layout features: the relative positions of each element on the canvas, such as center, top, bottom, left, right, and the occlusion or proximity relationship between elements;

[0041] Geometric size ratio: the proportion of a single target to the total area of ​​the canvas, and the size ratio between different targets;

[0042] Line quality: Whether there are abnormal brushstroke characteristics such as repeated smudging, partial blackening, broken lines, shaky lines, or excessive redrawing.

[0043] The above information is compressed into a fixed-dimensional image feature vector. It should be understood that this vector preserves both the projective semantics of the image content and computational operability.

[0044] (2) Input the dynamic handwriting trajectory data (a sequence of sampling points arranged in time sequence) into the temporal network model. Considering the time dependence and variable length sequence characteristics of handwriting dynamics, this embodiment uses a Long Short-Term Memory (LSTM) network or a Temporal Convolutional Network (TCN) as the encoder. The model processes the sampling points frame by frame along the time axis and outputs a dynamic feature vector of fixed length.

[0045] Within the model, a series of psychologically significant statistics are first calculated from the original trajectory, including but not limited to: instantaneous velocity sequence. , and its mean, variance, peak value and skewness; pen pressure sequence Mean, variance, maximum and minimum values, and number of pressure mutations; number of pauses and duration of a single pause (pause is defined as a period of time in which the speed is below a preset threshold and the displacement change is minimal in multiple consecutive sampling points); brush color switching order and switching frequency.

[0046] The aforementioned statistics only reflect the local or global aggregation characteristics of the trajectory and cannot fully describe the deep temporal dependencies hidden in handwriting. Therefore, while receiving the above statistics, the temporal network model further learns dynamic patterns directly from the original trajectory sequence, including the nonlinear coupling changes between speed and pressure, the connection rhythm between strokes, the transitional form of the strokes, and the fluctuation law of trajectory curvature over time.

[0047] The model jointly encodes the statistics and the temporal dependencies of the original sequence through loop or convolution operations, fusing them into a high-dimensional dynamic feature vector. It should be understood that this vector can effectively characterize the unconscious emotional fluctuations (such as high anxiety corresponding to a sudden increase in pressure), impulse control level (such as the correlation between pause frequency and speed change) and psychomotor excitement or inhibition state (such as the positive and negative coupling trend between overall speed and pressure) that users unconsciously reveal during writing and drawing.

[0048] (3) Input the written text content (Unicode string) into the natural language processing model. In this embodiment, a pre-trained language model (such as BERT or RoBERTa) is used to segment, embed, and encode the text. The model outputs a high-dimensional text semantic vector.

[0049] Optionally, to highlight explicit information relevant to psychological testing, a fine-grained analysis of the text semantic vector is performed to extract the following sub-features:

[0050] Emotional polarity scoring: The model outputs a continuous score on the positive-negative dimension of the text through the classification head, as well as the probability distribution of basic emotions such as anger, anxiety, and sadness.

[0051] Topic clustering feature: Map text word vectors to a predefined topic space (such as "work pressure", "family relationships", "self-evaluation", "future uncertainty", etc.) to obtain the weight of each topic;

[0052] Cognitive appraisal features: Identify whether the text contains absolute words (such as "always", "never"), negative expressions (such as "no", "not"), and the use of first-person singular and plural forms. These linguistic features are closely related to the user's cognitive style and level of self-focus.

[0053] Ultimately, the text feature vector It preserves the user's conscious language expression in a densely embedded form.

[0054] S3, perform cross-modal semantic conflict detection on the image features, the dynamic features and the text features. When a semantic conflict is detected, use orthogonal projection to perform semantic decoupling to extract the conflict feature matrix representing subconscious contradictions and the consistency feature matrix representing conscious collaboration. Integrate the conflict feature matrix and the consistency feature matrix into multimodal features.

[0055] To address the conflict between conscious and subconscious meaning caused by users' psychological defense mechanisms, this step first detects the existence of conflict. If conflict exists, semantic decoupling is performed through orthogonal projection to separate consistent and conflicting components, thereby preventing conflict signals from being simply averaged or filtered. The specific process is as follows:

[0056] As an example, cross-modal semantic conflict detection is performed on the image features, the dynamic features, and the text features, including:

[0057] S31, the image features, the dynamic features and the text features are mapped to a unified joint representation space to obtain the corresponding image semantic vector, dynamic semantic vector and text semantic vector respectively;

[0058] because , , Features from different encoding networks (e.g., ResNet, LSTM, BERT) have varying dimensions, numerical distributions, and semantic abstraction levels in their feature spaces. Specifically, image features tend to express spatial structure and visual semantics, dynamic features express temporal behavioral patterns, and text features express the meaning of linguistic symbols. Directly calculating or fusing their similarities can lead to biases due to inconsistent metrics. Therefore, this step first maps all three features to a unified joint representation space.

[0059] Three independent mapping networks are constructed, each implemented using a multilayer perceptron (MLP) containing one or two hidden layers and a non-linear activation function (such as ReLU). It should be understood that these mapping networks are used to project the original feature vectors into the same low-dimensional dense vector space, with an output dimension of [missing information]. (In this embodiment, we take) The mapping process can be represented as:

[0060]

[0061] in: As an image semantic vector, it encodes the projective features of the painting content, such as symbols, spatial layout, and blackening, into a semantic embedding with uniform dimensions;

[0062] As a dynamic semantic vector, it integrates dynamic patterns such as pen pressure fluctuation, speed change, and pause distribution, and transforms them into an embedded representation comparable to image semantics;

[0063] As a text semantic vector, it compresses the explicit emotions, themes, and cognitive evaluations of written text into embeddings in the same space.

[0064] The parameters of the three mapping networks are jointly optimized through end-to-end training, one of the training objectives being to ensure that in normal samples without psychological defenses, and and In the joint space, a high cosine similarity is maintained; while in samples with defenses, directional divergence is allowed.

[0065] It should be understood that, through the above mapping, the geometric heterogeneity between the three types of features is eliminated.

[0066] S32, calculate the first correlation degree between the text semantic vector and the image semantic vector, and the second correlation degree between the text semantic vector and the dynamic semantic vector;

[0067] In psychometrics, when users possess psychological defense mechanisms, their conscious verbal expressions (such as "I'm fine") often differ from their subconsciously driven drawing content and handwriting behavior. This invention utilizes text semantic vectors... It is considered a conscious benchmark because it directly originates from the user's handwritten text and is most subject to the user's subjective control; while image semantic vectors... and dynamic semantic vectors This reflects more of the subconscious projection. Users find it difficult to consciously control the pressure and speed of each stroke during free drawing and writing, and it is also more difficult to completely disguise all the symbolic features in the whole painting.

[0068] To quantify the degree of such consistency or conflict, calculations were performed separately. and , and The semantic relevance between vectors is measured. This embodiment uses cosine similarity as the relevance metric because it is insensitive to the magnitude of the vectors and only focuses on directional similarity, making it suitable for measuring semantic direction consistency. The specific calculation formula is as follows:

[0069] First degree of relevance (text-image):

[0070]

[0071] Second degree of relevance (text-dynamic):

[0072]

[0073] in, The Euclidean norm of a vector; and The range of values ​​for all values ​​is [-1, 1]:

[0074] when When the value is close to 1, it indicates that the directions of the two vectors are almost identical, meaning that the semantics of the text are highly consistent with the semantics of the corresponding modality, and there is no significant conflict between the user's conscious and subconscious expressions; when... When the value is close to 0, it indicates that the two vectors are orthogonal or nearly orthogonal, semantically unrelated, and may exhibit deviations due to defensive measures; when... When the value is negative, it indicates that the two vectors are in opposite directions, and there is a clear opposition between conscious and subconscious awareness (for example, the word "happy" is written, but the figure in the picture has a sad expression or the handwriting shows negative characteristics such as high pressure and rapid trembling). Therefore, the above calculation results... and Used to characterize the degree of semantic consistency between text and drawing content, and between text and handwriting behavior.

[0075] S33, compare the first correlation degree and the second correlation degree with the corresponding preset consistency threshold respectively. If the first correlation degree and / or the second correlation degree are less than the corresponding preset consistency threshold, it is determined that there is a semantic conflict.

[0076] In obtaining and Next, it is necessary to determine whether the current user has significant semantic conflict, i.e., inconsistency between conscious and subconscious mind under psychological defense, based on preset judgment rules. For this purpose, two consistency threshold parameters are preset: text-image consistency threshold. Text-Dynamic Consistency Threshold It should be noted that these two thresholds were obtained during the model training phase through statistical analysis or learning from a large amount of sample data labeled with psychological defense tags (e.g., levels of concealment tendency labeled by clinical psychologists). The default settings are available. , However, the specific value can be adjusted according to the application scenario (such as lowering the threshold if high sensitivity is required for clinical screening, and raising the threshold if high specificity is required), and will not be elaborated further.

[0077] like and If no significant cross-modal semantic conflict is found, it indicates that the semantic direction of the user's written text, drawing content, and handwriting dynamics is basically consistent. The user may currently be in a low psychological defense state, or their true emotions may match their verbal expression. In this case, semantic decoupling is unnecessary, and the three semantic vectors can be directly concatenated and fed into the subsequent psychological profiling model.

[0078] like and / or If so, a conflict of meaning is determined. This includes the following situations:

[0079] Only text and image conflicts ( but For example, if a user writes "a warm home", but the drawn house has no windows and the roof is painted black, the semantics of the image are clearly inconsistent with the text, while the handwriting may not show any obvious abnormalities.

[0080] Text-only and dynamic conflicts ( but For example, if a user writes "I am calm", but the handwriting shows an extremely high average stress level and drastic speed fluctuations, then the dynamic features reveal anxiety.

[0081] Both are in conflict ( and ): The most common high-defense scenario is when the text content is completely different from the drawing and handwriting.

[0082] After determining that there is a semantic conflict, the subsequent semantic decoupling branch (i.e., steps S34 to S36) is triggered to extract the conflict feature matrix and the consistency feature matrix in the form of orthogonal projection.

[0083] As an example, orthogonal projection is used for semantic decoupling to extract a conflict feature matrix representing subconscious contradictions and a consistent feature matrix representing conscious synergy, including:

[0084] S34, the image semantic vector and the dynamic semantic vector are matrix-combined to construct a feature matrix to be decoupled representing the subconscious, and the text semantic vector is constructed as a baseline feature matrix of conscious awareness;

[0085] Image features and dynamic features are inherently heterogeneous. Specifically, image features (such as houses, trees, figures, and their spatial layout in a painting) are essentially static global spatial features, and their encoding vectors... There is no fixed temporal order among the various dimensions; they reflect the spatial distribution and semantic relationships of elements on the user's final canvas. Dynamic features (such as pen pressure sequences, speed curves, and pause distributions) are essentially temporal features, and their encoded vectors... By extracting time-series networks such as LSTM or TCN, each dimension implicitly encodes the change pattern of the time dimension.

[0086] Clearly, static spatial features focus more on what was drawn and where it was placed, while temporal features focus more on how it was drawn and how emotions fluctuated. If we directly... and Simple matrix combinations (such as horizontal splicing) can lead to the numerical range of one modality dominating the combined matrix, while suppressing the effective information of another modality. Furthermore, different users reveal different levels of subconsciousness across different modalities; for example, some users' drawings are rich in content and have clear projections but lack dynamics in their handwriting, while others are the opposite. Simple combinations cannot distinguish which modality is currently more reliable. Additionally, directly combining low-quality or high-noise modal features will contaminate the subsequent decoupling results of orthogonal projection, and may even amplify the noise.

[0087] To address the aforementioned technical problems, this invention employs a dynamic combination method based on information entropy weighting, enabling the feature matrix to be decoupled to be adaptively weighted according to the degree of certainty of the actual information of each modality, thereby achieving a more robust cross-modal combination.

[0088] As an example, the image semantic vector and the dynamic semantic vector are matrix-combined to construct a feature matrix representing the subconscious, including:

[0089] S341, calculate the first information entropy of the image semantic vector to determine the first subconscious disclosure confidence level; and calculate the second information entropy of the dynamic semantic vector to determine the second subconscious disclosure confidence level.

[0090] To assess the current credibility of each modality's disclosure, information entropy is used as a metric. Information entropy reflects the degree of concentration of a probability distribution: the more concentrated the distribution (with a few dimensions dominating), the lower the entropy value, indicating that the modality's features are clearly defined and the confidence of its subconscious disclosure is high; the more uniform the distribution, the higher the entropy value, indicating that the features are vague or chaotic and the confidence is low.

[0091] Specifically, for image semantic vectors Each component is then converted into a probability distribution using the softmax function:

[0092]

[0093] in, express The Each component.

[0094] The formula for calculating the first information entropy is:

[0095]

[0096] It is a very small positive number (e.g., 10). -8 To avoid divergence in logarithmic operations. The theoretical scope is .when When the probability is close to 0, it indicates that the probability of one dimension is close to 1, while the probability of other dimensions is close to 0. This means that the image features are highly concentrated in a certain semantic direction, indicating that the user's drawing content has a clear projection direction (e.g., a very clear drawing of a large tree), and the confidence level of the subconscious expression is high; when near When the probability distribution is uniform, the content of the painting is blurry or contains a lot of irrelevant noise, indicating a low confidence level.

[0097] The entropy value is converted into the confidence level of the first subconscious disclosure using a negative exponential mapping function. :

[0098]

[0099] in, For example, take the adjustment factor. .

[0100] Similarly, for dynamic semantic vectors Calculate its corresponding probability distribution Second information entropy and confidence level of second subconscious expression .

[0101] Through the above calculations, the two quantitative confidence scores, namely the subconscious disclosure confidence scores, were obtained, which respectively characterize the reliability of the user's subconscious disclosure in the two channels of static drawing and dynamic handwriting.

[0102] S342, based on the first subconscious disclosure confidence level and the second subconscious disclosure confidence level, assign corresponding first feature weights and second feature weights to the image semantic vector and the dynamic semantic vector respectively, to obtain a weighted image semantic vector and a weighted dynamic semantic vector;

[0103] The confidence level obtained above and As feature weights, multiply them onto the corresponding semantic vector:

[0104]

[0105] After the above multiplication operation, the high-confidence mode receives a high weight: when the information entropy of a certain mode is low and the feature set is clear. When the value is close to 1, the weighted vector almost retains its original value, and this mode plays a dominant role in subsequent combinations. Meanwhile, low-confidence modes receive low weights: when a mode has high information entropy, fuzzy features, or high noise, As the vector approaches zero, the weighted vector is significantly compressed, effectively suppressing its interference with subsequent decoupling.

[0106] In this way, the contribution level can be dynamically adjusted according to the actual data quality of different users and modalities, avoiding the problem of low-quality features contaminating high-quality features in static splicing. For example, if the user's drawing content is rich and the direction is clear ( (High), but the handwriting is flat and unchanging. If the weighted average is low, then the weighted average will be low. The dominant feature matrix to be decoupled, and The contribution of static spatial features is suppressed; conversely, the contribution of temporal features is suppressed. The above adaptive method can effectively solve the problem of granularity imbalance between static spatial features and temporal features.

[0107] S343, The weighted image semantic vector and the weighted dynamic semantic vector are concatenated to form a matrix to construct the feature matrix to be decoupled.

[0108] After weighting, the weighted image semantic vector and the weighted dynamic semantic vector are combined into a matrix. For example, horizontal stitching (stitching by column) can be used to form a matrix. The matrix:

[0109]

[0110] Each column of this matrix corresponds to a weighted semantic vector of a subconscious modality, which is the feature matrix to be decoupled.

[0111] At the same time, the text semantic vector Constructing a baseline feature matrix for conscious awareness (Single column matrix), serving as the reference direction for subsequent orthogonal projection.

[0112] S35, the feature matrix to be decoupled is orthogonally projected onto the space spanned by the conscious reference feature matrix, the parallel projection component in the direction of the conscious reference feature matrix is ​​calculated, and the parallel projection component is determined as the consistent feature matrix;

[0113] Each column represents the embedding of a subconscious modality (image or handwriting) in the joint semantic space, while This represents the embedding direction of conscious textual expression. Furthermore, this step aims to decompose the subconscious signal into two components: one parallel (consistent) with the conscious direction, and the other orthogonal (conflicting), as follows:

[0114] exist In the semantic space, the conscious awareness benchmark matrix Zhang becomes a one-dimensional subspace, which is composed of unit vectors. Zhang Cheng. For any column vector (Right now (a column in) The orthogonal projection on the spanned subspace is defined as:

[0115]

[0116] It should be understood that this projection vector has the following properties: (and) The direction is parallel (i.e., consistent with the conscious reference direction); it is exist Distance among all vectors in the direction The nearest point; residual and Orthogonal.

[0117] Apply the above operations simultaneously The two columns can be efficiently implemented using matrix operations. First, calculate the projection coefficient matrix. :

[0118]

[0119] in, The result is a 2×1 column vector, with each component corresponding to a column of projection coefficients.

[0120] Therefore, the consistent feature matrix is:

[0121]

[0122] , its first The list is No. Parallel projection components listed in the conscious reference direction.

[0123] It should be noted that, at the psychological level, the consistent feature matrix This represents the information in subconscious signals that is consistent with the semantics of conscious text. For example, when a user writes "I am happy" and their drawing also includes positive elements such as sunshine and smiling faces, the components in the image and dynamic features that are consistent with the semantic direction of "happiness" will be retained in the consistency feature matrix. Even if the user has some defensiveness, some genuine positive emotions may still be unconsciously expressed through drawing. This part of the signal does not conflict with the text and should be used normally by the model.

[0124] S36, by subtracting the consistent feature matrix from the feature matrix to be decoupled, the orthogonal complement component orthogonal to the explicit conscious reference feature matrix is ​​calculated, and the orthogonal complement component is determined as the conflict feature matrix.

[0125] After obtaining the consistent feature matrix, the residual components orthogonal to the conscious reference direction are extracted by matrix subtraction. Specifically:

[0126]

[0127] The subtraction operation is performed element by element. Each column is an orthogonal complement component corresponding to a subconscious mode.

[0128] According to the definition of orthogonal projection The following orthogonality conditions must be met:

[0129]

[0130] Right now Each column is related to the conscious baseline feature matrix column vectors Orthogonal. Geometrically, lie in The direction of the span (i.e.) In the orthogonal complement space of the direction. This component cannot be obtained from conscious textual semantics through any linear combination, therefore it represents a part of the information in the subconscious modality that is independent of, or even opposed to, conscious expression.

[0131] It should be understood that in psychological defense scenarios, This is a matrix of conflict features representing subconscious contradictions. For example, a user writes "I'm fine," but in their drawing, they repeatedly blacken the roof of a house and depict figures without hands or feet. These negative projective features are orthogonal (or even opposite) to the semantic direction of "fine," and are thus fully preserved. In the middle, the user writes "not nervous", but the dynamic handwriting features show that the pen pressure fluctuates drastically and the pauses are abnormally frequent. These high anxiety dynamic features are orthogonal to "not nervous" and are also included in the conflict feature matrix.

[0132] Through the above decomposition, the subconscious conflict signals that the user deliberately conceals can be separated from the conscious disguise signals, without mutual cancellation or being misjudged as noise as in traditional feature splicing.

[0133] Furthermore, the consistent feature matrix and the conflicting feature matrix are horizontally concatenated:

[0134]

[0135] This is the final multimodal feature, which contains four columns: weighted image consistency component, weighted dynamic consistency component, weighted image conflict component, and weighted dynamic conflict component.

[0136] S4. Obtain the user's assessment intent, input the assessment intent and the multimodal features into the psychological profile model, infer the multidimensional psychological state assessment results, and generate a psychological assessment report accordingly.

[0137] The system receives the user's actively input assessment intent via a user intent interaction interface. This interface can be designed as one of the following two forms or a combination thereof:

[0138] Quick Tag Button: A set of common assessment intent tags are preset next to the canvas interface, such as "Personality Exploration", "Emotional State", "Stress Cause Analysis", "Career Development Direction", "Interpersonal Relationship Assessment", etc., which users can click to select;

[0139] Free text input box: Users can input their concerns about psychological issues using natural language, such as "I want to know why I've been having trouble sleeping lately" or "What kind of job would be suitable for me in the future?" The free text is then converted into predefined intent category codes by a natural language understanding module (such as a BERT-based intent classifier).

[0140] Once the user specifies their assessment intent, that intent is transformed into an intent guidance vector. ,in This represents the total number of intent categories (using one-hot encoding or learnable embedding vectors). If the user does not specify an intent (i.e., selects "General Assessment"), then... Set it to the zero vector or a fixed default baseline vector.

[0141] Multimodal features Flattened into a one-dimensional vector Then with the intention-guided vector The data is then pieced together to form the input for the mental profile model:

[0142]

[0143] It should be noted that the mental profiling model is a multi-task deep learning network, such as... Figure 2 As shown, its architecture includes:

[0144] Shared feature extraction layer: Several layers of fully connected (FC) networks with batch normalization and Dropout are used to extract high-order joint features of the input;

[0145] Multi-task output branches: Based on different dimensions of psychological assessment, multiple independent output heads are set. Each output head can be a fully connected layer with an appropriate activation function. For example, personality dimensions (introversion / extroversion, emotional stability, openness, etc.) use Sigmoid to output a [0,1] score; the degree of psychological distress (stress level, anxiety tendency, depressive tendency) uses Softmax to output the severity level probability or direct regression continuous value; potential dilemma types (family conflict, career confusion, interpersonal sensitivity) use multi-label classification to output the probability of each dilemma.

[0146] The training data for the psychological profiling model consists of a large number of drawing samples and corresponding professional psychological assessment labels (provided by clinical psychologists or standardized scales). A multi-task joint loss function, such as a weighted sum of losses from each task, is used during training. During forward inference, the input... The data is fed into the model and, after forward propagation, yields a vector of multidimensional psychological state assessment results. ,in This is the sum of all output dimensions.

[0147] As an example, generating a psychological assessment report includes:

[0148] An adaptive solution strategy corresponding to the multidimensional psychological state assessment result is matched from a preset strategy library, and a psychological assessment report containing the multidimensional psychological state assessment result and the adaptive solution strategy is generated and output.

[0149] Specifically, based on the results of the multidimensional psychological state assessment, a psychological assessment report is automatically generated for the user, as follows:

[0150] The strategy library pre-stores intervention plans corresponding to different psychological state assessment results. Each record in the strategy library includes: a set of psychological state vector threshold ranges or prototype vectors, as well as corresponding solution strategy text and action suggestions. The types of solution strategies include, but are not limited to: self-help training (such as mindfulness breathing guidance, emotion diary templates, progressive muscle relaxation audio); behavior adjustment suggestions (such as increasing outdoor activities, adjusting work and rest schedules, and reducing social media use); professional referral prompts (such as suggesting seeking evaluation from a psychologist or psychiatrist); and lifestyle fine-tuning suggestions (such as improving diet and increasing social contact).

[0151] Based on the evaluation result vector Using nearest neighbor search (such as calculating) Matching with the Euclidean distance of each prototype vector in the policy library or with a rule-based decision tree, select one or more of the most suitable solutions.

[0152] The final report is preferably presented in a graphic format, and may include the following:

[0153] Visualized multidimensional radar chart: Multiple indicators such as personality dimensions and degree of psychological distress are plotted into a radar chart to intuitively show the user's relative position in each dimension;

[0154] Textual interpretation: The meaning of each score is explained in language that is easy for users to understand. Combined with the subconscious contradiction information extracted from the conflict feature matrix, it points out the user's possible inconsistency or defensiveness, as well as potential psychological distress.

[0155] Adaptive solution strategies: The matched solution strategies are presented as a list of bullet points or step cards, along with actionable instructions (such as audio links or practice plan templates).

[0156] Reference Figure 3 As shown, the present invention also provides a multimodal psychological assessment system 100 based on drawing and handwriting, the system comprising:

[0157] The data acquisition module 10 is used to collect multimodal data during the user's drawing and writing operations on the interactive digital drawing board. The multimodal data includes static image data of the drawing, dynamic handwriting trajectory data, and written text content data.

[0158] Feature extraction module 20 is used to extract image features, dynamic features and text features from the static painting image data, the dynamic handwriting trajectory data and the written text content data, respectively;

[0159] The conflict detection and decoupling module 30 is used to perform cross-modal semantic conflict detection on the image features, the dynamic features and the text features. When a semantic conflict is detected, orthogonal projection is used to perform semantic decoupling to extract the conflict feature matrix representing subconscious contradictions and the consistency feature matrix representing conscious coordination. The conflict feature matrix and the consistency feature matrix are then integrated into multimodal features.

[0160] The reasoning report module 40 is used to obtain the user's assessment intent, input the assessment intent and the multimodal features into the psychological profile model, reason to obtain the multidimensional psychological state assessment result, and generate a psychological assessment report accordingly.

[0161] As an example, such as Figure 4 As shown, the conflict detection and decoupling module 30 includes:

[0162] The spatial mapping unit 301 is used to map the image features, the dynamic features and the text features to a unified joint representation space to obtain the corresponding image semantic vector, dynamic semantic vector and text semantic vector respectively.

[0163] The association calculation unit 302 is used to calculate the first association degree between the text semantic vector and the image semantic vector, and the second association degree between the text semantic vector and the dynamic semantic vector;

[0164] The conflict determination unit 303 is used to compare the first correlation degree and the second correlation degree with the corresponding preset consistency threshold respectively. If the first correlation degree and / or the second correlation degree is less than the corresponding preset consistency threshold, it is determined that there is a semantic conflict.

[0165] As an example, continue to refer to Figure 4 The conflict detection and decoupling module 30 further includes:

[0166] The matrix construction unit 304 is used to matrixally combine the image semantic vector and the dynamic semantic vector to construct a feature matrix to be decoupled representing the subconscious, and to construct the text semantic vector as a conscious baseline feature matrix.

[0167] The spatial projection unit 305 is used to orthogonally project the feature matrix to be decoupled onto the space spanned by the conscious reference feature matrix, calculate the parallel projection component in the direction of the conscious reference feature matrix, and determine the parallel projection component as the consistent feature matrix.

[0168] The difference calculation unit 306 is used to calculate the orthogonal complement component orthogonal to the explicit reference feature matrix by subtracting the consistent feature matrix from the feature matrix to be decoupled, and to determine the orthogonal complement component as the conflict feature matrix.

[0169] As an example, such as Figure 5 As shown, the matrix construction unit 304 includes:

[0170] The confidence determination subunit 3041 is used to calculate the first information entropy of the image semantic vector to determine the first subconscious disclosure confidence; and to calculate the second information entropy of the dynamic semantic vector to determine the second subconscious disclosure confidence.

[0171] The weight allocation subunit 3042 is used to allocate corresponding first feature weights and second feature weights to the image semantic vector and the dynamic semantic vector according to the first subconscious disclosure confidence and the second subconscious disclosure confidence, respectively, to obtain a weighted image semantic vector and a weighted dynamic semantic vector.

[0172] The matrix splicing subunit 3043 is used to perform matrix splicing of the weighted image semantic vector and the weighted dynamic semantic vector to construct the feature matrix to be decoupled.

[0173] As an example, the reasoning report module 40 is specifically used for:

[0174] An adaptive solution strategy corresponding to the multidimensional psychological state assessment result is matched from a preset strategy library, and a psychological assessment report containing the multidimensional psychological state assessment result and the adaptive solution strategy is generated and output.

[0175] This invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the preceding claims.

[0176] This invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the preceding claims.

[0177] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to the above embodiments without departing from the principles and spirit of the present invention, and all such changes, modifications, substitutions, and variations fall within the protection scope of the present invention.

Claims

1. A multimodal psychological assessment method based on drawing and handwriting, characterized in that, Includes the following steps: Collect multimodal data during the user's drawing and writing operations on the interactive digital drawing board, including static image data of drawing, dynamic handwriting trajectory data, and written text content data; Image features, dynamic features, and text features are extracted from the static image data of the painting, the dynamic handwriting trajectory data, and the written text content data, respectively. Cross-modal semantic conflict detection is performed on the image features, dynamic features and text features. When a semantic conflict is detected, orthogonal projection is used to decouple the semantics to extract the conflict feature matrix representing subconscious contradictions and the consistency feature matrix representing conscious coordination. The conflict feature matrix and the consistency feature matrix are then integrated into multimodal features. The system obtains the user's assessment intent, inputs the assessment intent and the multimodal features into the psychological profile model, infers the multidimensional psychological state assessment results, and generates a psychological assessment report accordingly.

2. The multimodal psychological assessment method based on drawing and handwriting as described in claim 1, characterized in that: Cross-modal semantic conflict detection is performed on the image features, the dynamic features, and the text features, including: The image features, dynamic features, and text features are mapped to a unified joint representation space to obtain the corresponding image semantic vector, dynamic semantic vector, and text semantic vector, respectively. Calculate the first correlation degree between the text semantic vector and the image semantic vector, and the second correlation degree between the text semantic vector and the dynamic semantic vector; The first correlation degree and the second correlation degree are compared with the corresponding preset consistency thresholds. If the first correlation degree and / or the second correlation degree are less than the corresponding preset consistency threshold, it is determined that there is a semantic conflict.

3. The multimodal psychological assessment method based on drawing and handwriting as described in claim 2, characterized in that: Semantic decoupling is performed using orthogonal projection to extract conflict feature matrices representing subconscious contradictions and consistent feature matrices representing conscious synergy, including: The image semantic vector and the dynamic semantic vector are matrix-combined to construct a feature matrix representing the subconscious, and the text semantic vector is used to construct a baseline feature matrix of conscious awareness. The feature matrix to be decoupled is orthogonally projected onto the space spanned by the conscious reference feature matrix, and the parallel projection component in the direction of the conscious reference feature matrix is ​​calculated. The parallel projection component is then determined as the consistent feature matrix. By subtracting the consistent feature matrix from the feature matrix to be decoupled, the orthogonal complement component orthogonal to the explicit conscious reference feature matrix is ​​calculated, and the orthogonal complement component is determined as the conflict feature matrix.

4. The multimodal psychological assessment method based on drawing and handwriting as described in claim 3, characterized in that: The image semantic vector and the dynamic semantic vector are matrix-combined to construct a feature matrix representing the subconscious, including: Calculate the first information entropy of the image semantic vector to determine the first subconscious disclosure confidence level; and calculate the second information entropy of the dynamic semantic vector to determine the second subconscious disclosure confidence level. Based on the first subconscious disclosure confidence level and the second subconscious disclosure confidence level, the image semantic vector and the dynamic semantic vector are respectively assigned corresponding first feature weights and second feature weights to obtain the weighted image semantic vector and the weighted dynamic semantic vector. The weighted image semantic vector and the weighted dynamic semantic vector are concatenated into a matrix to construct the feature matrix to be decoupled.

5. The multimodal psychological assessment method based on drawing and handwriting as described in claim 1, characterized in that: Generate a psychological assessment report, including: An adaptive solution strategy corresponding to the multidimensional psychological state assessment result is matched from a preset strategy library, and a psychological assessment report containing the multidimensional psychological state assessment result and the adaptive solution strategy is generated and output.

6. A multimodal psychological assessment system based on drawing and handwriting, characterized in that, The system includes: The data acquisition module is used to collect multimodal data during the user's drawing and writing operations on the interactive digital drawing board. The multimodal data includes static image data of the drawing, dynamic handwriting trajectory data, and written text content data. The feature extraction module is used to extract image features, dynamic features, and text features from the static image data of the painting, the dynamic handwriting trajectory data, and the written text content data, respectively. The conflict detection and decoupling module is used to perform cross-modal semantic conflict detection on the image features, the dynamic features and the text features. When a semantic conflict is detected, orthogonal projection is used to perform semantic decoupling to extract the conflict feature matrix representing subconscious contradictions and the consistency feature matrix representing conscious coordination. The conflict feature matrix and the consistency feature matrix are then integrated into multimodal features. The reasoning report module is used to obtain the user's assessment intent, input the assessment intent and the multimodal features into the psychological profile model, reason to obtain the multidimensional psychological state assessment results, and generate a psychological assessment report accordingly.

7. A multimodal psychological assessment system based on drawing and handwriting as described in claim 6, characterized in that: The conflict detection and decoupling module includes: The spatial mapping unit is used to map the image features, the dynamic features, and the text features to a unified joint representation space to obtain the corresponding image semantic vector, dynamic semantic vector, and text semantic vector, respectively. The association calculation unit is used to calculate the first association degree between the text semantic vector and the image semantic vector, and the second association degree between the text semantic vector and the dynamic semantic vector; The conflict determination unit is used to compare the first correlation degree and the second correlation degree with the corresponding preset consistency thresholds respectively. If the first correlation degree and / or the second correlation degree are less than the corresponding preset consistency thresholds, it is determined that there is a semantic conflict.

8. A multimodal psychological assessment system based on drawing and handwriting according to claim 7, characterized in that: The conflict detection and decoupling module further includes: The matrix construction unit is used to matrix-combine the image semantic vector and the dynamic semantic vector to construct a decoupled feature matrix representing the subconscious, and to construct the text semantic vector as a conscious baseline feature matrix. A spatial projection unit is used to orthogonally project the feature matrix to be decoupled onto the space spanned by the conscious reference feature matrix, calculate the parallel projection component in the direction of the conscious reference feature matrix, and determine the parallel projection component as the consistent feature matrix. The difference calculation unit is used to calculate the orthogonal complement component that is orthogonal to the explicit reference feature matrix by subtracting the consistent feature matrix from the feature matrix to be decoupled, and to determine the orthogonal complement component as the conflict feature matrix.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images