Drawing psychoanalysis system, method and device based on multi-mode interpretable intelligence

Abstract

The invention discloses a multi-modal interpretable intelligent drawing psychoanalysis system, which comprises an image acquisition module used for acquiring user drawing data and facial expression data; the feature extraction module is used for extracting area, position and shadow features from the drawing data; the classification module is used for classifying the extracted features based on a support vector machine (SVM) in combination with a radial basis kernel function so as to distinguish positive and negative psychological states; the interpretability analysis module is used for generating visual interpretation through class activation mapping and labeling and quantifying psychological characteristics on which the network decision depends in combination with predefined psychological symbols; and the knowledge graph reasoning module is used for carrying out psychological state reasoning by utilizing a Transform model based on the constructed drawing psychological knowledge graph. According to the method, an efficient and credible intelligent analysis tool is provided for psychological experts, meanwhile, non-intrusive and convenient psychological assessment service is provided for subjects, and drawing psychological analysis is promoted to be transformed from experience driving to data driving.

Description

Technical Field

[0001] The present invention relates to the field of psychoanalysis, and particularly to a painting psychoanalysis system, method and device based on multi-modal interpretable intelligence. Background Art

[0002] Existing painting psychoanalysis technologies have significant defects at the hardware and software levels:

[0003] Hardware limitations:

[0004] Single function: Traditional high-speed scanners only support single-channel image acquisition and cannot synchronously obtain multi-modal data (such as paintings and facial expressions).

[0005] Design defects: The device has a large volume, and the front-protruding optical elements invade the painting space of the subject, interfering with the operation; and it lacks the ability of embedded algorithm processing and relies on external computing devices.

[0006] Poor portability: The device is difficult to fold and store, which limits the flexibility of on-site psychological general surveys.

[0007] Software limitations:

[0008] Model black box: The painting classification model based on deep learning (such as CNN) lacks interpretability and cannot be associated with the psychological knowledge system, resulting in psychological experts being difficult to trust the results.

[0009] Difficult cross-domain migration: General image description models rely on large-scale labeled data, and there are significant style differences between the training set and the painting psychology dataset (such as HTP), so the direct migration effect is poor.

[0010] Lack of knowledge reasoning: Existing technologies have not constructed a painting psychology knowledge graph and cannot perform deep psychological state reasoning through a semantic network.

[0011] Insufficient data alignment: There are significant distribution differences between the source domain (general dataset) and the target domain (HTP dataset), and the model generalization ability is limited.

[0012] Core problems: The single hardware function, poor software interpretability, weak cross-domain generalization ability, and insufficient knowledge reasoning ability lead to low efficiency of psychoanalysis and insufficient credibility of the results.

[0013] Therefore, those skilled in the art have provided a painting psychoanalysis system, method and device based on multi-modal interpretable intelligence to solve the problems raised in the above background art. Summary of the Invention

[0014] To solve the above technical problems, the present invention provides a painting psychoanalysis system based on multi-modal interpretable intelligence, including:

[0015] An image acquisition module for obtaining user painting data and facial expression data;

[0016] A feature extraction module for extracting area, position, and shadow features from the painting data;

[0017] A classification module that classifies the extracted features based on the support vector machine (SVM) combined with the radial basis kernel function to distinguish between positive and negative mental states;

[0018] An interpretability analysis module that generates visual explanations through class activation mapping and quantifies the mental features on which the network decision depends by combining predefined mental symbol annotations;

[0019] A knowledge graph reasoning module that performs mental state reasoning using the Transformer model based on the constructed painting psychology knowledge graph.

[0020] Preferably: In the feature extraction module:

[0021] The area feature is obtained by calculating the total area of the target boxes in the painting;

[0022] The position feature is obtained by calculating the average of the center point coordinates of all target boxes;

[0023] The shadow feature is realized by dividing the painting into multiple Patches and calculating the pixel density of each Patch.

[0024] Preferably: In the classification module:

[0025] A convex quadratic programming problem is constructed using slack variables and the radial basis kernel function, and the classification of non-linearly separable data is achieved by maximizing the classification boundary and penalizing misclassified samples.

[0026] Preferably: The interpretability analysis module further includes:

[0027] A mental symbol annotation unit that annotates 9 types of mental symbols including house, tree, person, sun, cloud, dog, bird, chimney, and apple;

[0028] A feature quantization unit that calculates the total area within the class and the center point position of the mental symbols to generate an interpretability report associated with psychological knowledge.

[0029] Preferably: It further includes a sketching processing module for:

[0030] Converting the input painting image into a sketch style to reduce the style difference between the training data and the test data;

[0031] Optimizing the image quality through binarization and grayscale processing.

[0032] Preferably: The knowledge graph reasoning module includes:

[0033] The knowledge graph construction unit extracts entities and relationships based on the drawing psychological projection rule base to construct a triple semantic network;

[0034] The context subgraph sampling unit generates a local graph structure input sequence through the central triple and its neighborhood triples;

[0035] The structure-enhanced Transformer architecture combines the adjacency matrix to encode graph structure information to achieve link prediction of heterogeneous knowledge graphs.

[0036] Preferably: The knowledge graph reasoning module further adopts a masked knowledge modeling mechanism:

[0037] Randomly mask the tokens in the input sequence and complete the knowledge graph completion by predicting the masked tokens;

[0038] Combine the BERT pre-trained model to initialize the encoding task and enhance the semantic understanding ability.

[0039] Preferably: It further includes a multi-modal feature conversion module for:

[0040] Adopt the CLIP model as the image encoder to extract visual embedding features;

[0041] Map the visual features to the semantic space of the GPT-2 text decoder through a mapping network;

[0042] Introduce a manifold regularization term to constrain feature alignment and improve the generalization of cross-domain data.

[0043] Preferably: The multi-modal feature conversion module further includes a domain adaptation unit:

[0044] The parallel domain discriminator distinguishes the source domain and target domain features and combines the gradient reversal layer to confuse the domain differences;

[0045] Realize the migration of the training model to the target domain through local geometric structure preservation and minimum distribution difference.

[0046] Preferably: The image acquisition module supports dynamic face tracking:

[0047] Real-time locate the face position through the Haar cascade classifier;

[0048] Generate a control signal based on the position difference to adjust the servo to ensure the quality of expression data acquisition.

[0049] Preferably: The system further includes a perspective correction module:

[0050] Identify the four corner points of the painting image through the foot point detection algorithm;

[0051] Generate a standard frontal image using an image transformation matrix to eliminate tilt interference.

[0052] A computer-readable storage medium stores computer program instructions that, when executed by a processor, implement all the functions of the system according to any one of claims 1-11.

[0053] A method for painting psychological analysis includes the following steps:

[0054] Collect user painting and facial expression data;

[0055] Extract area, position, and shadow features;

[0056] Classify positive and negative psychological states through an SVM classifier;

[0057] Generate a visual explanation based on class activation mapping and combine psychological symbols to quantify decision-dependent features;

[0058] Use knowledge graph reasoning to generate a description of the psychological state.

[0059] Preferably: The knowledge graph reasoning step further includes:

[0060] Perform sketch preprocessing on the painting;

[0061] Generate an initial text description through the CLIP and GPT-2 models;

[0062] Perform knowledge graph link prediction based on the Relphormer framework and output a psychological assessment result.

[0063] Preferably: The method further includes a domain adaptation step:

[0064] Constrain the feature alignment between the source domain and the target domain through a manifold regularization term;

[0065] Combine masked knowledge modeling to enhance the model's reasoning ability for heterogeneous data.

[0066] A foldable multi-modal data acquisition hardware device includes:

[0067] An intelligent terminal with an embedded computing platform for running data preprocessing and analysis algorithms;

[0068] A foldable arm connected to the intelligent terminal through a base, and the arm includes:

[0069] A lower arm equipped with a facial expression sampling module including a camera and a servo control component;

[0070] An upper arm equipped with a painting sampling module with an inclined plane image correction function;

[0071] V-shaped spring energy storage component, connecting the lower arm and the upper arm, providing elastic energy storage in the folded state and stability in the unfolded state;

[0072] Locking mechanism for fixing the folded or unfolded state of the support arm.

[0073] Preferably, the facial expression sampling module further includes:

[0074] Haar cascade classifier for real-time detection of the face position and calculation of the difference in the picture boundary;

[0075] GPIO port for outputting a duty cycle square wave signal to control the servo to adjust the camera angle and ensure dynamic tracking accuracy.

[0076] Preferably, the painting sampling module includes:

[0077] Foot point detection algorithm for identifying the coordinates of the four corner points of the painting image;

[0078] Image transformation matrix generation unit for generating a standard front-view image based on the corner point coordinates to eliminate tilt interference.

[0079] Preferably, the base is designed as:

[0080] With a heat dissipation channel inside, communicating with the heat dissipation structure of the smart terminal;

[0081] The outer contour matches that of the smart terminal to ensure that the center of gravity coincides when the support arm is unfolded and avoid the risk of tipping.

[0082] Preferably, the V-shaped spring energy storage component:

[0083] Is locked by a tenon and stores elastic potential energy in the folded state;

[0084] Automatically releases the potential energy after the tenon is released, driving the support arm to unfold to a preset angle.

[0085] Preferably, the locking mechanism includes:

[0086] Mechanical buckle for fixing the folded state of the support arm;

[0087] Electromagnetic unlocking unit for releasing the buckle after receiving an external instruction to achieve one-key unfolding.

[0088] Preferably, the smart terminal adopts the NVIDIA Jetson series embedded computing platform and integrates the following functions:

[0089] Real-time processing of facial tracking and painting correction algorithms;

[0090] Stores the collected multi-modal data (images, videos) and analysis results.

[0091] Preferably, when the hardware device is in the deployed state:

[0092] The lower arm and the upper arm form an angle of 90° to 120°, ensuring that there is no occlusion between the face and the painting sampling field of view;

[0093] The overall weight is evenly distributed, and the portability meets the needs of on-site psychological general surveys.

[0094] The present invention significantly improves the efficiency and reliability of painting psychological analysis through multi-modal interpretable intelligent technology. The specific effects are as follows:

[0095] Hardware innovation:

[0096] Multi-modal data fusion: Synchronously collect painting image and facial expression data, reducing the operation complexity and improving the data integrity.

[0097] Portability and non-invasiveness: The foldable arm design supports one-key deployment and storage, facilitating on-site deployment; the dynamic tracking technology (servo adjustment) avoids interfering with the subjects.

[0098] Software innovation:

[0099] High-precision classification: Based on the classification model of SVM and overall features (area, position, shadow), the accuracy rate reaches 98%, effectively distinguishing positive and negative psychological states.

[0100] Enhanced interpretability: Generate visual interpretation maps through class activation mapping (CAM), combined with quantitative analysis of psychological symbols (such as house, tree, person), mapping the model decision to the psychological knowledge system, and providing decision-making basis understandable by experts.

[0101] Knowledge graph reasoning: Construct the first house-tree-person painting psychological knowledge graph (HTP_KG), and use the Relphormer framework to achieve semantic reasoning, filling the gap in the field.

[0102] Cross-domain generalization ability: The sketching process reduces the data style differences, combined with manifold regularization and domain adaptation technology, significantly improving the migration effect of the model on foreign data.

[0103] Systematic advantages:

[0104] End-to-end closed-loop: Form a complete analysis chain from data collection, feature extraction, classification to knowledge reasoning, reducing manual intervention.

[0105] Scalability: The modular design supports subsequent function expansion (such as adding new psychological symbol categories or knowledge graph relationships).

[0106] Comprehensive effect: The present invention provides an efficient and reliable intelligent analysis tool for psychological experts, and at the same time provides a non-invasive and convenient psychological assessment service for the subjects, promoting the transformation of painting psychological analysis from experience-driven to data-driven. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] Figure 1 FIG. is a schematic diagram of the combination of the support arm and the terminal of the hardware device of the present application;

[0108] Figure 2 FIG. is a structural diagram of the hardware device of the present application (in the storage state);

[0109] Figure 3 FIG. is a structural diagram of the hardware device of the present application (in the unfolded state);

[0110] Figure 4 FIG. is a flowchart of the face tracking algorithm of the system of the present application;

[0111] Figure 5 FIG. is a flowchart of the perspective correction algorithm of the system of the present application;

[0112] Figure 6 FIG. is a schematic diagram of the painting psychological image emotion screening of the system of the present application;

[0113] Figure 7 FIG. is the original image and the image after shadow representation of the system of the present application;

[0114] Figure 8 FIG. is a model framework diagram of the system of the present application;

[0115] Figure 9 FIG. is a flowchart of the algorithm of the sketching module of the system of the present application;

[0116] Figure 10 FIG. is a word segmentation schematic diagram of the construction part in the knowledge graph reasoning module of the system of the present application;

[0117] Figure 11 FIG. is a schematic diagram of the construction part in the knowledge graph reasoning module of the system of the present application;

[0118] Figure 12 FIG. is a schematic diagram of the overall framework in the knowledge graph reasoning module of the system of the present application;

[0119] Figure 13 FIG. is a schematic diagram of the system flow of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0120] The present invention will be further described in detail below in conjunction with the drawings and specific embodiments. The embodiments of the present invention are given for purposes of illustration and description, and are not intended to be exhaustive or to limit the invention to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are chosen and described in order to best explain the principles of the invention and its practical application, and to enable those of ordinary skill in the art to understand the invention and design various embodiments with various modifications suitable for specific purposes.

[0121] Please refer to Figures 1 to 13 , in this embodiment, a painting psychological analysis system based on multimodal interpretable intelligence is provided.

[0122] (1) Hardware Deployment, Data Acquisition and Processing

[0123] Please refer to Figures 1 to 3 , to implement the above functions, we provide an integrated hardware design that can achieve multi-channel multimedia sampling and data preprocessing.

[0124] The hardware consists of a foldable sampling arm that can achieve multi-channel multimedia sampling and an intelligent terminal equipped with the required related algorithm models. The arm structure consists of a base connecting the lower arm and the terminal, a lower arm equipped with a facial expression sampling module, an upper arm equipped with a painting sampling module, a folding energy storage component based on a V-shaped spring, and a locking mechanism. The intelligent terminal consists of an NVIDIA Jetson series embedded computing platform, peripheral components, and a housing.

[0125] In terms of the arm, as Figure 3 shown, the base mainly serves to fix the terminal and the lower arm. Considering the heat dissipation design of the terminal and the avoidance of related components, the overall arm does not affect the heat dissipation inside the intelligent terminal when covering the intelligent terminal and the centers of gravity of the two coincide. The lower arm and the base are connected by two concentric short shafts with clearance fit. The lower arm is provided with a bracket for placing the facial expression sampling module supported by two servos. After the sampled image data enters the intelligent terminal, the position of the human face is obtained through a pre-trained Haar cascade classifier. By comparing with the position of the picture boundary, the relative position difference is calculated and a square wave with a specified duty cycle is output from the GPIO port to control the servo to adjust in the specified direction.

[0126] The upper arm is equipped with a painting sampling module. Four corner points of the inclined plane painting image are obtained through a foot point detection algorithm trained with a dedicated dataset. After sorting the four point coordinates, they are sent into an image transformation matrix and applied to the image to achieve image transformation. The transformed image is a standard front view image.

[0127] The short shaft is equipped with a V-shaped spring to maintain the unfolded state of the arm through tension and is locked by a tenon in the folded state. At this time, the V-shaped spring stores energy. The two states are as Figure 2 、 3 shown. Removing the tenon can unfold this hardware at once.

[0128] (2) Feature Extraction Module

[0129] In this design, the required related algorithms and models are loaded in the above intelligent terminal. The following is an introduction to each algorithm model.

[0130] AsFigures 6 to 7 As shown, in the positive and negative classification part of the painting psychological screening function, the overall features are the most important evaluation factors. In this part of the algorithm, three overall painting features, namely area, position, and shadow, are key selected for negative painting screening. The process is as follows Figure 4 As shown, first, a feature extractor is used to extract key features such as the area, position, and shadow of the picture, and these features are combined to construct a painting classifier. This painting classifier can effectively perform emotional screening on painting psychological images, screen out negative paintings for further diagnosis and treatment.

[0131] Among them, the area refers to the total area of all objects in the painting, and the overall area of the painting is calculated by the total area of the marked target boxes. A total of 9 objects are labeled, including houses, trees, people, the sun, birds, chimneys, clouds, dogs, and fruits.

[0132] The research uses Backbone and RoIpooling as fixed feature extractors to extract features such as area, position, and shadow. The area of the picture is:

[0133]

[0134] s∈R 1*1 . λ is the number of target boxes in the painting. W0 is the width of the target box. h0 is the height of the target box. W is the width of the painting. H is the height of the painting.

[0135] The position of the picture, that is, the center point of the picture, is represented by the coordinates of the center points of all target boxes on a painting.

[0136]

[0137] p∈R 2*1 . Taking the lower left corner of the painting as the coordinate origin. m0 is the abscissa of the center point of the target box, and n0 is the ordinate of the center point of the target box.

[0138] Shadow generally refers to large-area smearing. The method of using split Patches is selected to calculate the pixel density to represent the shadow and achieve feature extraction of the shadow. First, the painting is binarized:

[0139]

[0140] θ is the binarization threshold. g(u,v) is the pixel value of the grayscale image at the point (u,v), where g w indicates that the pixel value of this point is 255, which is a white pixel. Then the painting is cropped into 16 Patches, and the white pixel density of each Patch is calculated:

[0141]

[0142] q belongs to R 2*1 。w p is the width of the Patch, h p is the height of the Patch. If the pixel density in the Patch is large, as shown in Equation 1-2, it indicates that there is a shadow in the Patch.

[0143] Connect the features extracted by the two feature extractors to form a 19-dimensional feature vector q ∈ R 19*1 。The connection of multiple features such as area, position, and shadow integrates information from different feature extractors, thus creating a more representative and discriminative feature representation.

[0144] Through this method, the most important feature in the painting analysis process, that is, the overall feature, is successfully obtained, focusing on important features such as area, position, and shadow in the subsequent classification task, thereby realizing the screening of effective negative paintings.

[0145] (III) Classification Module

[0146] Based on the relationship between the overall feature and the mental state, paintings are divided into two categories: positive and negative. When completing the classification task, a Support Vector Machine (SVM) classifier is used for classification. This method extracts the overall feature of the painting, associates it with the mental state, realizes the classification of the painting, and further distinguishes its positive and negative mental states.

[0147] Framework of the automatic painting screening method:

[0148] As a powerful classifier, the support vector machine can effectively handle non-linear relationships. The input is D = {(x1, y1)(x2, y2),...,(x n , y n )}, where x i ∈ R 19*1 , y i ∈ (0, 1), i = 1, 2,..., n, and N is the number of the painting. x i is the overall feature vector, and y i is the label of the painting. As non-linearly separable data, slack variables C and kernel function k(x, x i ) are introduced. Construct and solve the convex quadratic programming problem:

[0149]

[0150] 0 ≤ α i ≤ C, i = 1, 2,..., N

[0151] Among them, the first part attempts to maximize the boundary between classes, while the second part aims to penalize samples located on the incorrect side of the hyperplane. The optimal solution is:

[0152]

[0153] α that satisfies 0 < α j * < C * components, and calculate the bias:

[0154]

[0155] Therefore, the classification decision function is:

[0156]

[0157] The k(x, x i ) used is the radial basis function,

[0158]

[0159] where γ is a parameter that controls the width of the Gaussian kernel. The accuracy of the SVM in classifying the dataset depends on the parameters C and γ. Therefore, for this algorithm, the values of C and γ must be carefully determined. In practice, the values of C and γ can be determined by control variables.

[0160] This part of the algorithm relies on the relationship between mental states and painting features, constructs a mental state classifier by extracting overall features, and realizes painting classification.

[0161] (IV) Interpretability Analysis Module

[0162] In the interpretable part of the painting psychological screening function, traditional interpretable methods based on class activation mapping reveal the important regions of the input HTP drawing on which the intelligent painting psychological analysis model's decision depends, but fail to clearly reveal the connection between the network decision and the knowledge in the field of painting psychology. Therefore, this part of the algorithm proposes an interpretable painting psychological analysis method based on mental feature quantification, and explores the potential connection between the network decision and the field of painting psychology by analyzing mental features such as the area and position of mental symbols. The algorithm architecture is as Figure 8 shown. First, use the interpretable method of class activation mapping to generate a basic visual interpretation of the painting psychological model. Secondly, segment the basic visual interpretation through the annotation file related to mental symbols to obtain the mental symbols of the model decision. Finally, conduct a quantitative analysis of the position and area mental features of the mental symbols on which the model decision depends to explore the connection between the network decision and the knowledge in the field of painting psychology.

[0163] To narrow the gap between the features relied on by network decision-making and the psychological symbols in the field of painting psychology, 9 types of important symbols related to psychology (house, tree, person, sun, cloud, dog, bird, chimney, apple) were marked for each sample. These symbols carry specific psychological and emotional meanings in the field of painting psychology. In this paper, through this kind of marking, the features relied on by network decision-making were combined with the existing psychological knowledge system to obtain the psychological symbols relied on by network decision-making.

[0164] In painting psychological analysis, psychological experts usually conduct evaluations from three levels, namely overall analysis, analysis of the drawing process, and analysis of drawing details. Among them, the overall analysis of the painting is the most important evaluation criterion, mainly analyzing the attributes such as the position and size of the overall painting. For example, the picture size is too large, too small, or the position is deviated. This algorithm uses the analysis rules of psychological symbols in the field of painting psychology to evaluate the professional field attributes of the psychological symbols relied on by network decision-making, mainly from two aspects: area and position, aiming to improve the professional interpretability of the importance of network decision-making in the field of painting psychology, so that psychological experts can better understand and apply the output results of the model.

[0165] Each psychological feature area relied on by network decision-making is the area of psychological symbols highlighted by the interpretable algorithm, and the darker the color of the area, the stronger the importance for the decision made by the network. Therefore, it is necessary to extract the psychological symbol areas with strong importance. By setting a color threshold range and comparing each pixel of the psychological symbols relied on by network decision-making in each sample with the color threshold, the pixel a within the color threshold range ij is a part of the important area of the psychological symbols relied on by network decision-making. By calculating the number of pixels left within the color threshold in each sample, it approximates the area S of the area with strong importance of the psychological symbols relied on by network decision-making in each sample gc , as follows:

[0166]

[0167]

[0168] In the formula, p ij is the RGB value of each pixel in the visual interpretation, and φ is the attention threshold of the important area of the psychological symbol, which is a color range in the RGB space.

[0169] Each sample may have multiple important psychological symbols, and the number of each type of psychological symbol is also different. According to the different types of psychological symbols, the areas of the same type of psychological symbols in the sample are summed to obtain the total within-class area S of each type of psychological symbol related to network decision-making in the sample l , as follows:

[0170]

[0171] In the formula, S gc is the area of the mental symbol upon which each network decision in each sample depends. g ∈ [1, 2, …, m] is the number of each type of mental symbol related to network decision in each sample, and m is the maximum value of the number of each type of mental symbol related to network decision in the sample.

[0172] Sum the areas of the important regions of mental symbols for all categories to obtain the total area S of mental symbols related to network decision in the whole graph total , as follows:

[0173]

[0174] In the formula, S l is the separate total area of each type of mental symbol related to network decision. l ∈ [1, 2, …, λ] is the type of mental symbol existing in the sample, and the maximum value of λ is 9.

[0175] The position attribute of mental features is another important attribute for mental feature evaluation. This part of the algorithm extracts the important regions of mental symbols with strong importance that the network focuses on by setting an attention threshold. Take the center point position of the extracted important region of the mental symbol as the reference basis for the position attribute of the mental feature. However, most of the extracted important regions of mental symbols are irregular regions, so fit a minimum circumscribed rectangle to the region of the extracted mental symbol. Calculate the four coordinates (x, y, w, h) of the minimum circumscribed rectangle, where (x, y) is the center point coordinate of the rectangle and is used as the approximate position coordinate P gc (x, y) of the important region of the mental symbol, as follows:

[0176] P gc (x, y) = Rect(p ij ∈ φ) = (x, y)

[0177] In the formula, p ij is the RGB value of each pixel in the visual interpretation, and φ is the attention threshold of the important region of the mental symbol. Rect(·) is the function to find the minimum circumscribed rectangle of the region, and the minimum circumscribed rectangle (x, y, w, h) of the region of the mental symbol upon which each network decision depends can be obtained by this method. x, y, w, h represent the abscissa, ordinate, width, and height of the rectangle center respectively.

[0178] There may be multiple important mental symbols in each sample, and the number of each type of mental symbol is also different. According to the different types of mental symbols, take the average of the center point coordinates of the mental symbols of the same category in the sample as the within-class position coordinate P of the important region of each type of mental symbol related to network decisionl (x, y) is as follows:

[0179]

[0180] In the formula, P gc (x), P gc (y) are the horizontal and vertical coordinates of the minimum bounding rectangle of each psychological symbol region on which the network decision depends in each sample. g ∈ [1, 2, …, m] is the number of each type of psychological symbol related to the network decision in each sample, and m is the maximum value of the number of each type of psychological symbol related to the network decision in the sample.

[0181] Taking the average of the central point position coordinates of the important regions of all types of psychological symbols, the total central point coordinates P total (x, y) of the psychological symbols related to the network decision in the sample are obtained as follows:

[0182]

[0183] In the formula, P l (x), P l (y) are the total intra-class position coordinates of each type of psychological symbol related to the network decision. l ∈ [1, 2, …, λ] is the type of psychological symbol existing in the sample, and the maximum value of λ is 9.

[0184] In terms of the painting psychological assessment function:

[0185] The purpose of this step is to overcome the problem that there is a large difference between the training dataset (open-source general dataset) and the test dataset (collected painting images) in the pre-training model method of image description, and provide an automatic image description generation method that can reduce the dependence on the correlation between training data and test data and achieve knowledge or model transfer.

[0186] To achieve the above purpose, the technical solution of this step is as follows:

[0187] (1) Sketching processing module

[0188] The painting images collected in the previous step belong to sketch-style images. For the images in the general dataset used for training, there are problems in image style and quality, such as blurring, low resolution, and mismatch with the style of the collected house-tree-person test set images, which will have a greater impact on subsequent painting reasoning and image description. Therefore, this method first performs sketching processing on the dataset and uses the processed dataset to participate in subsequent model training. The process of the preprocessing part is as Figure 9 shown.

[0189] (2) Conversion from visual features to text features (multi-modal feature conversion module)

[0190] This method uses a pre-trained CLIP model as the image encoder and the open-source Chinese pre-trained large language model GPT-2 as the text decoder. The input image x i passes through the pre-trained image encoder to obtain the visual embedding feature vector f i = CLIP(x i ). Then, through the mapping network F, the visual embedding feature vector is converted into an encoded embedding feature vector that can be understood by GPT These sequences are input into the subsequent language model, and finally a meaningful text description is generated.

[0191] The training process is shown in Equation 2-1. Among them, l is the maximum length of the marker sequence padding, and θ represents the trainable parameters of the model. We train the model to learn the distribution from images to text descriptions by maximizing the log-likelihood probability of the training set text markers for the input picture x i

[0192]

[0193] The key component of this part is the mapping network. Since the Chinese semantic space is more complex, therefore, we introduce local structure preservation in the visual-to-text feature alignment, mine the deep information of the data, and achieve cross-modal alignment of features.

[0194] Specifically, we add a manifold regularization term on the basis of the mainstream cross-entropy function to constrain the mapping network to learn a domain-invariant mapping from image features to text features, and achieve image descriptions of unlabeled cross-domain data. The training loss is shown in Equation 2-2. Among them, L y is the total loss, L c is the cross-entropy loss, L r is the manifold regularization term. Here, the p-order Laplacian operator is taken as an example, and β is the regularization weight.

[0195]

[0196] (3) Image feature alignment between the source domain and the target domain (multi-modal feature conversion module)

[0197] Although we have performed operations to make the image styles consistent, it is obviously far from enough to achieve accurate migration. There are still significant differences in the image styles of the two data sets. For this reason, we also introduce local geometric structures, mine the deep connections of the data, adopt the idea of domain adaptation, reduce the data distribution differences between the training set and the test set, and achieve the transfer application of the trained image description model in the painting image test set.

[0198] In domain adaptation, we refer to the auxiliary dataset with sufficient annotations as the source domain, that is, the general dataset after sketch processing, and the dataset with few or no annotations as the target domain (or called the destination domain), that is, our House-Tree-Person painting set. Our ultimate goal is to transfer the knowledge learned on the source domain to the target domain through methods such as feature alignment, so that the unannotated images in the target domain can also use the model trained on the source domain.

[0199] We paralleled a domain discriminator in the above task framework to distinguish whether the extracted encoded embedding feature vectors come from the source domain or the target domain. To achieve the consistency of the two-domain features and make the discriminator confused about the vector source, the loss should be maximized, so a gradient reversal layer is added to be consistent with the minimization of the other part of the loss.

[0200] The role of local structure is mainly manifested in the following two aspects:

[0201] ① Integration during training: As a regularization means during the training process, it helps the model better understand and handle the similarities and differences between the source domain and the target domain. In this way, the model can more effectively learn how to adapt to the characteristics of the target domain while maintaining the source domain information.

[0202] ② Transfer between domains: Measure and minimize the difference between the data distributions of the two domains. As a domain adaptation mechanism, it helps the model learn how to transfer the features learned in the source domain to the target domain while maintaining the structural consistency of the two domains.

[0203] (V) Knowledge Graph Reasoning Module

[0204] For the actual problem of generating painting psychological descriptions, we propose an interpretable psychological description method based on knowledge graph reasoning. Entities, attributes, and the relationships between entities are extracted from the painting psychological projection rule base to construct a House-Tree-Person painting psychological knowledge graph, and through a reasoning method based on Transformer, the psychological description is inferred from the obtained painting interpretations, thus realizing painting psychological analysis.

[0205] A. Knowledge Graph Construction

[0206] (1) Knowledge Word Segmentation

[0207] Word segmentation is the process of splitting long text into independent words, where each word contains independent meaning. This

[0208] process can split the semantics of the text into finer units, making the text structure simpler and easier for the computer to understand and analyze. Using a Chinese word segmentation tool based on Python for word segmentation mainly consists of three steps. First, use regular expressions to roughly split the Chinese paragraph into individual sentences, such as:

[0209] Original sentence: [Large doors and windows on the house often mean that the painter actively contacts the outside world and pursues to be understood.]

[0210] After segmentation: [“, ’Large doors and windows on the house’, ‘often mean that the painter actively contacts the outside world’, ’pursues to be understood’,”]

[0211] Secondly, construct a directed acyclic graph for each sentence, and then find the best segmentation scheme. Construct a directed acyclic graph (DAG) according to the clauses segmented in Step 1, as Figure 10 shown.

[0212] Use dynamic programming to calculate the best word segmentation method and select the global best scheme.

[0213] Finally, for consecutive single characters, use the Hidden Markov Model (HMM) to

[0214] segment them again. The final word segmentation is as follows:

[0215]

[0216] Through word segmentation, the collected text is segmented into sentences to obtain the drawing psychological projection rule base.

[0217] (2) Knowledge graph construction

[0218] Construct a knowledge graph of house-tree-person drawing psychology according to the drawing psychological projection rules. The construction process is as follows Figure 11 shown.

[0219] Through operations such as knowledge extraction and entity alignment, extract drawing descriptions and psychological descriptions as entities, and the relationship

[0220] between them as a relationship to form triples and construct a knowledge graph of house-tree-person drawing psychology.

[0221] B. Realization of reasoning for the knowledge graph of drawing psychological analysis

[0222] This section will introduce the overall model framework specifically used, and use the Relphormer[D] method to complete this process, including the initialization part, context subgraph sampling, Transformer backbone network, and masked knowledge modeling. For the initialization part, use the BERT[D10] pre-trained model for the encoding task; for the backbone network part, due to the fully connected nature of the attention mechanism in Transformer, the structural information of non-sequential input may be lost, so a structure-enhanced attention mechanism is adopted, and masked knowledge modeling is used to mask entities or relationships in the knowledge graph triples to complete the training and testing tasks.

[0223] (1) Overall framework

[0224] In this paper, a Transformer-based framework is mainly used for the link prediction task of the painting psychology knowledge graph. Its overall framework is as follows Figure 12 As shown, among them, the content on the far left is the context sampling part, the middle part is the self-attention mechanism with enhanced structure, and the right part is the masked knowledge modeling. Among them, the sequence of the context subgraph sampling is directly input into the Transformer model, while the adjacency matrix is used in the structure encoder.

[0225] (2) Context subgraph sampling strategy

[0226] The painting psychology knowledge graph contains a large amount of relationship information. An entity usually connects different relationships, making the knowledge graph a large-scale heterogeneous network. However, the entire graph structure cannot be directly fed into the Transformer model architecture. Therefore, a context subgraph sampling strategy is used to encode local graph structure information through the contextualized subgraph input sequence. The context subgraph T G consists of a central triple T center and its neighborhood triple T n :

[0227] T G = T center ∪ T n (0-1)

[0228] The sampling of the subgraph is carried out at the triple level, and the neighborhood triple T context is obtained through the sampling of the subgraph:

[0229] T n = {T | T i ∈ N} (0-2)

[0230] where N is the fixed-size domain triple of T center .

[0231] Through the context subgraph sampling, the subgraph T G of the central entity context can be obtained. At the same time, the local structure of the context subgraph is retained in the adjacency matrix A G , and its expression form is:

[0232] A G = {a uv} ∈ {0, 1} |V|×|V| (0-3)

[0233] where a uvFor the interaction between node pairs (u, v), where the relationship represented by the edge is regarded as a special entity, so (u, v) can also be the interaction between an entity and a relationship. When the node pair (u, v) is connected, a uv takes the value of 1, otherwise 0.

[0234] (3) Structure-enhanced Transformer architecture

[0235] The Transformer model can capture global information well. A knowledge graph is a graph structure whose nodes are not arranged in order. The nodes in the knowledge graph can be located in a multi-dimensional space and are connected by edges. In order to encode the structural information of the graph in the model, a structure-enhanced Transformer is adopted under the basic Transformer model, aiming to retain the important structural information in the context subgraph. Specifically, attention bias is used to capture the structural information between node pairs in the context subgraph. The attention bias is denoted as φ(i, j), which is the bias term between node v i and node v j . The specific formulas are shown as follows (3-5) and (3-6).

[0236]

[0237] is the normalized adjacency matrix, W Q and W K are the query vectors in the Transformer module, h i and h j are the hidden representations, and d is the hidden dimension. The structure encoder f structure is a linear layer with as the input, where m is a hyperparameter. refers to the reachable relevance obtained by taking m steps from one node to another node. In this paper, the adjacency matrix is concatenated as the overall structural information.

[0238] (4) Masked knowledge modeling

[0239] Based on the Transformer architecture, this part of the algorithm uses the masked knowledge modeling mechanism for prediction. It randomly masks specific tokens in the input sequence and then predicts these masked tokens. Only one token is randomly masked for a sequence to better integrate context information and finally obtain the corresponding result.

[0240] Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art and related fields based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention. The structures, devices, and operation methods not specifically described and explained in the present invention shall be implemented by conventional means in the art without special instructions and limitations.

Claims

1. A painting psychological analysis system based on multimodal interpretable intelligence, characterized in that, Including: An image acquisition module for acquiring user painting data and facial expression data; A feature extraction module for extracting area, position, and shadow features from the painting data; A sketch processing module for converting the input painting image into a sketch style to reduce the style difference between training data and test data, and optimizing the image quality through binarization and grayscale processing. A multi-modal feature conversion module that uses the CLIP model as an image encoder to extract visual embedding features, maps the visual features to the semantic space of the GPT-2 text decoder through a mapping network, introduces a manifold regularization term to constrain feature alignment, and improves the generalization of cross-domain data; A classification module that classifies the extracted features based on a support vector machine (SVM) combined with a radial basis kernel function to distinguish positive and negative mental states; An interpretability analysis module that generates visual explanations through class activation mapping and quantifies the mental features on which the network decision depends by combining predefined mental symbol annotations; A knowledge graph reasoning module that performs mental state reasoning using the Transformer model based on the constructed painting psychology knowledge graph.

2. The system according to claim 1, wherein In the feature extraction module: The area feature is obtained by calculating the total area of the target boxes in the painting; The position feature is obtained by calculating the average value of the center point coordinates of all target boxes; The shadow feature is realized by dividing the painting into multiple Patches and calculating the pixel density of each Patch.

3. The system according to claim 1, characterized in that, In the classification module: A convex quadratic programming problem is constructed using slack variables and a radial basis kernel function, and the classification of non-linearly separable data is achieved by maximizing the classification boundary and punishing misclassified samples.

4. The system according to claim 1, wherein The interpretability analysis module further includes: A mental symbol annotation unit that annotates 9 types of mental symbols including houses, trees, people, suns, clouds, dogs, birds, chimneys, and apples; A feature quantification unit that calculates the total area within the class and the center point position of the mental symbols, and generates an interpretability report associated with psychological knowledge.

5. The system according to claim 1, wherein The knowledge graph reasoning module includes: A knowledge graph construction unit that extracts entities and relationships based on the painting psychology projection rule library and constructs a triple semantic network; A context subgraph sampling unit that generates a local graph structure input sequence through the central triple and its neighborhood triples; A structure-enhanced Transformer architecture that combines an adjacency matrix to encode graph structure information and realizes link prediction for heterogeneous knowledge graphs; The knowledge graph reasoning module further adopts a masked knowledge modeling mechanism: Randomly mask the tokens in the input sequence and complete the knowledge graph completion by predicting the masked tokens; Combine the BERT pre-trained model to initialize the encoding task and enhance the semantic understanding ability.

6. The system according to claim 1, wherein The multi-modal feature conversion module further includes a domain adaptation unit: A parallel domain discriminator distinguishes source domain and target domain features and combines a gradient reversal layer to confuse domain differences; Through local geometric structure preservation and minimum distribution difference, the trained model is migrated to the target domain.

7. The system according to claim 1, wherein The image acquisition module supports dynamic face tracking: The position of the human face is located in real time through a Haar cascade classifier; A control signal is generated based on the position difference to adjust the servo motor to ensure the quality of expression data acquisition.

8. The system according to claim 1, wherein The system further includes a perspective correction module: Identify the four corner points of the painting image through the foot point detection algorithm; Apply the image transformation matrix to generate a standard front view image and eliminate the tilt interference.

9. A method for painting psychological analysis, characterized in that, It includes the following steps: Collect the user's painting and facial expression data; Extract area, position, and shadow features; Conduct positive and negative mental state classification through the SVM classifier; Generate visual explanations based on class activation mapping and quantify decision-dependent features in combination with mental symbols; Use knowledge graph reasoning to generate a description of the mental state.

10. The method according to claim 9, wherein The knowledge graph reasoning step further includes: Preprocess the painting by sketching; Generate an initial text description through the CLIP and GPT-2 models; Perform knowledge graph link prediction based on the Relphormer framework and output the mental assessment results.

11. The method according to claim 10, characterized in that, The method further includes a domain adaptation step: Constrain the feature alignment between the source domain and the target domain through the manifold regularization term; Enhance the model's reasoning ability for heterogeneous data by combining masked knowledge modeling.

12. A foldable multi-modal data acquisition hardware device, characterized in that, It includes: An intelligent terminal with a built-in embedded computing platform for running data preprocessing and analysis algorithms; A foldable arm connected to the intelligent terminal through a base, and the arm includes: A lower arm equipped with a facial expression sampling module, including a camera and a servo control component; An upper arm equipped with a painting sampling module with an inclined plane image correction function; A V-shaped spring energy storage component connecting the lower arm and the upper arm, providing elastic energy storage in the folded state and stability in the unfolded state; A locking mechanism for fixing the folded or unfolded state of the arm; The facial expression sampling module further includes: A Haar cascade classifier for real-time detection of the face position and calculation of the picture boundary difference; A GPIO port for outputting a duty cycle square wave signal to control the servo to adjust the camera angle and ensure dynamic tracking accuracy; The painting sampling module includes: A foot point detection algorithm for identifying the four corner point coordinates of the painting image; An image transformation matrix generation unit for generating a standard front view image based on the corner point coordinates and eliminating the tilt interference; A heat dissipation channel is arranged inside the base, which is connected to the heat dissipation structure of the intelligent terminal, and the outer contour matches that of the intelligent terminal to ensure that the centers of gravity coincide when the arm is unfolded and avoid the risk of tipping; The V-shaped spring energy storage component is locked by a tenon and stores elastic potential energy in the folded state. After releasing the tenon, the potential energy is automatically released to drive the arm to unfold to a preset angle; The locking mechanism includes: A mechanical buckle for fixing the folded state of the arm; An electromagnetic unlocking unit for releasing the buckle after receiving an external instruction to achieve one-key unfolding.

Images