An abnormal psychological sand table detection method and system based on multi-modal contrast learning

Abstract

The application provides an abnormal psychological sand table detection method and system based on multi-modal contrast learning. The application solves the problem of single data mode by obtaining multiple modal data of the psychological sand table. Secondly, the self-weighting mechanism can dynamically adjust according to the modal self-consistency and cross-modal geometric redundancy, solving the problem of neglecting the dynamic contribution difference between modes. Finally, a joint optimization strategy combining intra-sample contrast learning and inter-sample contrast learning is adopted. Intra-sample contrast learning ensures the consistency between different modes of the same sample and maximizes the task relevance of high-quality modes. Inter-sample contrast learning makes semantically similar samples close to each other in the common representation space, and different samples far away from each other. The application learns more discriminative and robust multi-modal representations through the double consistency constraints of intra-sample and inter-sample, thereby significantly improving the accuracy and recall rate of abnormal psychological sand table recognition in actual scenes with scarce labeled data and unbalanced samples.

Description

Technical Field

[0001] This invention relates to the field of abnormal psychological detection technology, and in particular to an abnormal psychological sandplay detection method and system based on multimodal contrastive learning. Background Technology

[0002] Mental health issues have become a major global public health challenge. Traditional methods of mental health assessment, such as self-rating scales, expert interviews, and physiological indicator testing, have limitations such as poor effectiveness, low efficiency, and high cost. Sandplay therapy, as a non-verbal projection tool, allows individuals to unconsciously externalize their inner world by freely placing sandplay figures, providing a unique perspective for mental health assessment.

[0003] Currently, research on deep learning-based psychological sandplay analysis is still in its early stages. Most existing studies utilize only single-modal data, failing to integrate the rich information contained in the sandplay process, images, and symbolic meanings of the sandplay objects. This results in models capturing only a single dimension of information and having limited ability to discern complex psychological states. Therefore, multimodal learning research continues to deepen in the field of mental health assessment, revealing the following technical bottlenecks when applying multimodal self-supervised contrastive learning to psychological sandplay analysis:

[0004] The ability to depict complex psychological states is insufficient. The psychological states reflected in psychological sandplay often require multiple modalities to fully represent them. Existing methods only learn shared semantics by bringing different modal representations closer together, which makes it easy to miss or misjudge subtle or contradictory psychological states.

[0005] The fusion process exhibits poor adaptability. In sandplay works depicting different psychological states, the proportion of discriminative information carried by each modality varies significantly. Existing methods use static or equal-weighted approaches to process each modality, failing to dynamically adjust modal contributions based on specific sample content. This results in the discriminative signals of key modalities being overwhelmed by highly redundant information, leading to a decrease in the discriminative power of the fused representation.

[0006] The generalization ability is weak under sparse labeling conditions. High-quality labeling of mental sandplay relies on experts, which is extremely costly, and labeled samples are scarce. Existing methods only establish comparative relationships between different modalities or augmented views of the same sandplay, failing to utilize the potential semantic associations between unlabeled samples to assist representation learning. When labeling is severely insufficient, the model struggles to construct a globally discriminative feature space and lacks robustness to unseen anomalous patterns.

[0007] In view of this, the inventors have specifically designed an abnormal psychological sandplay detection method and system based on multimodal contrastive learning, which leads to this invention. Summary of the Invention

[0008] To solve the above problems, the technical solution of the present invention is as follows: An abnormal psychological sandplay detection method based on multimodal contrastive learning includes the following steps: Acquire multimodal data of psychological sandplay, including sandplay images, sandplay operation process, and semantic text of sandplay objects; Feature extraction is performed on the sand table image, the sand table operation process, and the semantic text of the sand table tools to obtain the corresponding image features, operation features, and semantic features. Data augmentation is performed on the sand table image, and the features extracted from the sand table operation process and the semantic text of the sand tools are enhanced to generate enhanced features corresponding to each modality; Based on the enhancement features of each modality, the self-consistency metric and cross-modal geometric redundancy metric of each modality are calculated. Based on the self-consistency metric and cross-modal geometric redundancy metric, a self-weighting mechanism is introduced to calculate the adaptive weight of each modality in the current sample. The image features, operational features, and semantic features are concatenated and a multimodal common representation is generated through a fusion network. The following two contrastive learning methods are then used to jointly optimize this multimodal common representation: Intra-sample contrastive learning: Align the enhanced features of different modalities of the same sample with their generated multimodal common representations, and align each single modal feature with its corresponding multimodal common representation. During the alignment process, the adaptive weights are used to weight the alignment strength of each modality. Inter-sample contrastive learning: Based on the similarity between the common representations of different samples in multimodal studies, an inter-sample relationship graph is constructed. Directly related sample pairs in the graph are used as positive samples, and unrelated sample pairs are used as negative samples. The common representations of positive sample pairs are aligned, while the common representations of negative sample pairs are separated. The optimized multimodal representation is input into a classifier to determine whether the target object's mental sandplay is an abnormal mental sandplay.

[0009] Preferably, the calculation of the adaptive weights for each mode specifically includes: For each modality, the cosine similarity between its original features and the corresponding enhanced features is calculated as a measure of the modality’s self-consistency. For each modality, the average similarity between its original features and the original features of other modalities is calculated as a measure of cross-modal geometric redundancy. Based on the modal self-consistency and cross-modal geometric redundancy, a weight score is calculated for each mode, wherein the weight score is positively correlated with the modal self-consistency metric and negatively correlated with the cross-modal geometric redundancy metric. The weight scores of all modes are normalized to obtain the adaptive weights for each mode.

[0010] Preferably, the inter-sample comparison learning includes: Construct a dynamic memory queue based on momentum updates to store the multimodal common representation of historical batch samples; Based on the multimodal common representation of the current batch of samples and all samples in the memory queue, the cosine similarity between samples is calculated, and the initial graph topology is constructed using the K-nearest neighbor sparsification strategy. On the K-nearest neighbor initial graph topology, sample pairs that are directly connected by an edge are used as positive sample pairs, and sample pairs that are unreachable in the multi-hop range are used as negative sample pairs. A graph contrastive learning loss function is constructed to optimize the multimodal common representation.

[0011] Preferably, the K-nearest neighbor initial graph topology uses the mutual K-nearest neighbor algorithm to prune the initial directed edges, retaining only bidirectional connected edges to form an undirected graph adjacency matrix.

[0012] Preferably, the classifier uses FocalLoss as the loss function during training.

[0013] Preferably, the sand table operation process is subject to feature extraction. Based on the operation time, operation type, operation object and spatial coordinate information in the process data, the emotional features in the operation process data are extracted from the time dimension, spatial dimension, operation dimension and sand table diversity dimension.

[0014] Preferably, the feature extraction of the semantic text of the sandplay includes constructing semantic features that reflect the overall positivity, emotional intensity, and thematic consistency of the sandplay based on a predefined sandplay emotional symbol library.

[0015] This invention also provides an abnormal psychological sandplay detection system based on multimodal contrastive learning, comprising: The multimodal data acquisition module is used to acquire multimodal data of the target object's psychological sandplay, including sandplay images, sandplay operation process, and semantic text of sandplay objects; The multimodal feature extraction module includes an image encoder, an operation feature encoder, and a semantic feature encoder, which are used to extract and enhance features from the sand table image, the sand table operation process sequence, and the emotional semantic information of the sand table tools, respectively, to obtain corresponding image features, operation features, and semantic features. The adaptive weight calculation module calculates the self-consistency metric and cross-modal geometric redundancy metric for each modality based on the enhancement features of each modality. Based on the self-consistency metric and cross-modal geometric redundancy metric, a self-weighting mechanism is introduced to calculate the adaptive weight of each modality in the current sample. The multimodal representation joint optimization module is used to generate a multimodal common representation based on the concatenated features of the image features, operational features, and semantic features; and to jointly optimize the multimodal common representation by applying intra-sample contrastive learning and inter-sample contrastive learning; wherein, the intra-sample learning is used to align the multimodal common representations generated by different enhanced versions of the same sample, and to use the adaptive weights to align each single-modal feature with its corresponding multimodal common representation; the inter-sample contrastive learning is used to construct a relationship graph based on the similarity between samples, aligning directly related sample pairs in the graph and separating unrelated sample pairs; The abnormal psychological sandplay classification module is used to input the optimized multimodal common representation into the classifier to determine whether the psychological sandplay of the target object is an abnormal psychological sandplay.

[0016] Preferably, the multimodal representation joint optimization module includes: The fusion unit is used to map the features of each modality to the same feature space as the common representation, and to identify the number of activated modalities with the help of the modality indicator vector, thereby generating the enhanced representation of each single modality and the global fusion representation; In-sample optimization unit, used to construct the in-sample contrastive learning loss function; The inter-sample optimization unit is used to construct the inter-sample contrastive learning loss function; The joint optimization unit is used to perform joint optimization of the multimodal common representation by weighting and combining the intra-sample contrast loss and the inter-sample contrast loss.

[0017] The technical solution provided by this invention has the following beneficial effects: This invention addresses the issue of single data modality by acquiring three types of data: sand table images, operational processes, and semantic text of sand tools, thus providing a data foundation for multimodal fusion. Secondly, it introduces a self-weighting mechanism, calculating adaptive weights based on the information characteristics of each modality in the current sample. These weights can be dynamically adjusted according to modal self-consistency and cross-modal geometric redundancy, solving the problem of ignoring dynamic contribution differences between modalities. Finally, it employs a joint optimization strategy combining intra-sample contrastive learning and inter-sample contrastive learning. Intra-sample contrastive learning ensures consistency between different modalities within the same sample and maximizes the task relevance of high-quality modalities, while inter-sample contrastive learning mines semantic associations between samples by constructing a dynamic graph structure, bringing semantically similar samples closer together in a common representation space and distancing different samples, thereby enhancing the discriminative and generalization capabilities of the representation. This invention can fully integrate complementary information in multimodal mental sandplay data, adaptively balance the contribution of each modality in the fusion process, and learn more discriminative and robust multimodal representations through dual consistency constraints inside and outside the sample. Thus, in real-world scenarios where labeled data is scarce and samples are imbalanced, it can significantly improve the accuracy and recall of identifying abnormal mental sandplays. Attached Figure Description

[0018] The accompanying drawings, which are provided to further illustrate the invention and constitute a part of this invention, are illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention.

[0019] in: Figure 1 This is a schematic diagram of the overall framework of the present invention. Detailed Implementation

[0020] To make the technical problems, solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of the invention.

[0021] Please refer to Figure 1 This invention provides an abnormal psychological sandplay detection method based on multimodal contrastive learning. The framework takes a sandplay image (I), a sandplay operation process (O), and semantic text of the sandplay objects (E) as inputs. After data augmentation and preprocessing, these are fed into modality-specific encoders to extract corresponding single-modality features. The modal features are concatenated along the channel dimension and then input into a fusion module to generate a unified common representation. The model is optimized at two levels: (a) intra-sample consistency optimization, which uses contrastive learning loss to constrain the consistency of the representation of the same sample across different modalities and their fused views; and (b) out-of-sample consistency optimization, which utilizes a momentum queue to construct a dynamic contrast graph and introduces a graph-based positive and negative sample relationship to further enhance the discriminativeness and robustness of the common representation.

[0022] Specifically, the following steps are included: Acquire multimodal data from psychological sandplay, including sandplay images, sandplay manipulation processes, and semantic text of sandplay objects; Feature extraction is performed on sand table images, sand table operation processes, and semantic text of sand table objects to obtain corresponding image features, operation features, and semantic features. This invention uses a pre-trained ResNet as the backbone network for image feature extraction to obtain high-level semantic visual representations, and incorporates color-related prior information into the feature extraction process to enhance the sensitivity of image representation to differences in emotion and style. The operation process data includes information such as operation timestamps, operation types, operation objects, and spatial coordinates. Emotional features in the operation process data are extracted from the time dimension, spatial dimension, operation dimension, and sand table object diversity dimension. Feature extraction of the sand table object semantic text includes constructing semantic features reflecting the overall positivity, emotional intensity, and thematic consistency of the sand table based on a predefined sand table object emotional symbol library.

[0023] Data augmentation is performed on the sand table images, and features extracted from the sand table operation process and semantic text of the sand table tools are enhanced to generate enhanced features corresponding to each modality; among which, data augmentation includes: Constructing an image dual-view transformer For the original image Generate the first view respectively With the second view First view Basic geometric transformations are used, including horizontal flattening, vertical flipping, and random erasure, while maintaining the scale (1.0, 1.0) and without scaling; Second View Automatic augmentation strategies and random scaling transformations are added to the first view to enhance data diversity; both views use the same interpolation method, mean normalization, and standard deviation normalization.

[0024] Constructing a two-view transformer for the operation process table For the original operation table data Generate the first view With the second view First view Z-score standardization ,in, The table's feature mean vector. The standard deviation vector of the table features. Numerical stable term; second view Add Gaussian noise to the standardization. ,in, This is the preset noise standard deviation.

[0025] Sentiment semantic table data Using the exact same enhancement strategy as the table data in the operation process, a dual view of semantic features is generated. .

[0026] The above modal dual views are aligned according to the samples to form a preprocessed and enhanced multimodal training sample set. .

[0027] Based on the enhancement features of each modality, a self-weighting mechanism is introduced to calculate the adaptive weight of each modality in the current sample. The adaptive weight refers to the normalized weight coefficient dynamically generated based on the modal self-consistency and cross-modal geometric redundancy presented by each modality in the current sample. Its value range is [0,1], and the sum of all modal weights is 1. The image features, operational features, and semantic features are concatenated and a multimodal common representation is generated through a fusion network. The following two contrastive learning methods are then used to jointly optimize this multimodal common representation: Intra-sample contrastive learning: Align the enhanced features of different modalities of the same sample with their generated multimodal common representations, and align each single modal feature with its corresponding multimodal common representation. During the alignment process, the adaptive weights are used to weight the alignment strength of each modality. Inter-sample contrastive learning: Based on the similarity between the common representations of different samples in multimodal studies, an inter-sample relationship graph is constructed. Directly related sample pairs in the graph are used as positive samples, and unrelated sample pairs are used as negative samples. The common representations of positive sample pairs are aligned, while the common representations of negative sample pairs are separated. The optimized multimodal representation is input into a classifier to determine whether the target object's mental sandplay is an abnormal mental sandplay.

[0028] This application also provides the following steps for calculating the adaptive weights for each mode: Given three modal features of a sample and its corresponding enhanced view The signs are consistent. We calculate a sample-level weight for each modality i. ∈[0,1], comprehensively measures modal self-consistency and cross-modal geometric redundancy.

[0029] The self-consistency of a modality is measured by the cosine similarity between its original features and the corresponding augmented view features, denoted as . This reflects the semantic stability of the modality under data augmentation; cross-modal geometric redundancy is defined as the average cosine similarity between the features of this modality and those of other modalities, i.e. A higher value indicates a greater overlap in the representation directions of different modes, and thus higher substitutability. Considering both modal self-consistency and cross-modal geometric redundancy, this application defines a self-weighted score for each mode i, as shown in the formula:

[0030] Where λ controls the redundancy suppression strength. The final weights are normalized using temperature-scaled softmax, and are expressed as follows:

[0031] in This is a temperature coefficient used to adjust the sharpness of the weight distribution.

[0032] These weights do not participate in feature fusion, but are used as sample-level importance factors to weight the loss in the subsequent multimodal consistency comparison loss. High weights correspond to semantically stable modalities, and their alignment signals are amplified in the total loss; conversely, the influence of low-quality or homogeneous modalities is automatically suppressed.

[0033] This application also provides intra-sample contrastive learning, with a loss function that narrows the distance between multimodal common representations generated by different augmented versions of the same sample, and narrows the distance between each single-modal feature and its corresponding multimodal common representation. When calculating the contrastive loss between single-modal features and common representations, corresponding adaptive weights are used for weighting. In intra-sample optimization, contrastive learning is performed between single-modal feature representations and common representations to retain the unique task-related information in each modality to the greatest extent possible, preventing the loss of modality-specific discriminative features during fusion. Contrastive learning is performed between common representations of different versions to maximize the retention of complex intermodal collaborative information and overall task-related information.

[0034] This application approximates the mutual information maximization objective through contrastive learning with augmented samples, and employs the InfoNCE mutual information estimator, as shown in the following formula: The intra-sample contrast loss incorporates adaptive weights, as shown below: On the one hand, the model performs comparative learning on the common representations of different enhanced versions, effectively capturing and preserving complex collaborative information between modalities and overall task-related information. On the other hand, the model performs comparative learning on each modal feature and its corresponding common representation, introducing task-related information from each modality, enhancing the unique information of each modality in the common representation, and considering that semantically stable modalities will be given higher weights in the contrastive loss.

[0035] Specifically, this application also provides inter-sample comparison learning, with the following steps: A dynamic memory queue based on momentum updates is constructed to store the multimodal common representation of historical batches of samples. This application introduces a dynamic memory queue to cache the fusion features of historical batches. This queue acts as a dynamic neighbor pool, retrieving semantically similar samples as potential neighbors when constructing the adjacency matrix of the current batch, thereby achieving cross-batch semantic interaction. Because the queue is continuously updated, the constructed context possesses temporal continuity and semantic evolution, and can be regarded as an implicit, dynamic approximation of the potential full graph structure.

[0036] Based on the multimodal common representation of the current batch samples and all samples in the memory queue, the cosine similarity between samples is calculated, and an initial graph topology is constructed using a K-nearest neighbor sparsification strategy. During training, this application constructs a dynamic graph topology in real-time based on the memory queue to achieve adaptive representation of latent semantic relationships. Given a set of nodes in the current batch and their historical cached features... (Where Q is the total number of nodes involved in graph construction). First, the features are... Normalize and calculate the cosine similarity matrix between nodes, as shown in the following formula:

[0037] The initial graph topology is constructed using a K-Nearest Neighbor (K-NN) sparsification strategy. For each node i, its K most similar neighbors are retained, forming an asymmetric adjacency matrix obtained according to formula (11).

[0038] in Let represent the set of indices of the K nodes with the highest similarity to node i. This strategy reduces computational complexity through local sparsity while preserving the semantically most relevant adjacency relationships.

[0039] On the initial K-nearest neighbor graph topology, sample pairs directly connected by edges are designated as positive sample pairs, and sample pairs unreachable within multiple hops are designated as negative sample pairs. A graph contrastive learning loss function is constructed to optimize the multimodal common representation. For anchor node i in the current batch, its positive samples are given by its first-order neighbors in the dynamic graph. Based on the binarized adjacency matrix... The set of positive samples can be defined as follows: These nodes have explicit and reliable relationships in the graph structure and should maintain similarity in the representation space. Negative sample construction is entirely based on the relationships between nodes. Reachability relationships. The model first generates a reachability matrix through matrix exponentiation. Defined as ,in express Adjacency matrix, Represents a node A node can be reached from node i via at most h hops. Therefore, a set of unreachable nodes can be defined. These nodes are structurally located in the far end of the graph, making it difficult for them to interact with anchor nodes through multi-hop paths, thus exhibiting high semantic discriminativeness.

[0040] The structural contrast constraint is defined using the InfoNCE loss, which enables the model to simultaneously maintain the aggregation of neighborhood features and the separation of distant structural regions in the structural space. Its contrast loss function is shown in the following formula:

[0041] in, For batch size, Let be the common representation of the i-th sample. This loss function simultaneously optimizes all positive sample pairs, encouraging anchor nodes to maintain high similarity with all their first-order neighbors in the representation space, while pushing unreachable nodes to the far end of the representation space. Compared to pairwise optimization of triplet loss, multi-positive sample loss can fully utilize graph structure information, avoid competition between positive samples, and provide a stronger supervision signal through global comparison of all negative samples in the denominator.

[0042] Specifically, the initial K-nearest neighbor graph topology uses the mutual K-nearest neighbor algorithm to prune the initial directed edges, retaining only bidirectionally connected edges to form the undirected graph adjacency matrix. To further improve the semantic consistency of the graph structure and suppress noise caused by unidirectional adjacency, this application adopts a mutual K-NN pruning strategy. Undirected edges are retained only when two nodes are each other's K-nearest neighbors, expressed as the following formula:

[0043] Through this two-way filtering mechanism, the dynamic graph topology retains only node pairs with strong semantic relevance and more stable structure.

[0044] Specifically, the proportion of anomalous samples in the psychological sandplay dataset is low, and the high cost of labeling leads to a scarcity of labeled samples. Traditional cross-entropy loss is easily affected by class imbalance and the dominance of a large number of easily distinguishable normal samples, making it difficult to effectively learn the discriminative boundary. This application adopts FocalLoss as the optimization objective in the fine-tuning stage. It dynamically reduces the weight of easily distinguishable samples by introducing a modulation factor, so that the model focuses on difficult-to-distinguish samples (especially minority class anomalous samples), as shown in the following formula:

[0045] in The probability of the model predicting the true class ∈[0,1] represents the class balancing weights, and γ≥0 is the focusing parameter, which controls the weight decay rate of easy and difficult samples. In the experiment, γ was set to 2 to enhance the focus on low-confidence outlier samples.

[0046] This invention also provides an abnormal psychological sandplay detection system based on multimodal contrastive learning, including... The multimodal data acquisition module is used to acquire multimodal data of the target object's psychological sandplay, including sandplay images, sandplay operation process, and semantic text of sandplay objects; The multimodal feature extraction module includes an image encoder, an operation feature encoder, and a semantic feature encoder, which are used to extract features from sand table images, sand table operation process sequences, and sand table emotional semantic information, respectively, to obtain corresponding image features, operation features, and semantic features. Specifically: The image encoder uses a pre-trained ResNet-50 backbone network, including convolutional layers, batch normalization layers, ReLU activation layers, max pooling layers, and four residual block groups; the last fully connected layer of ResNet-50 is removed, retaining only the global average pooling layer, with an output dimension of 2048; the operational feature encoder and semantic feature encoder are both multilayer perceptron (MLP) structures, including two encoding layers, each consisting of a linear transformation layer, a batch normalization layer, a ReLU activation layer, and a dropout layer; the first encoding layer takes the input feature dimension... Shrink to The second encoding layer reduces the feature dimension from Mapped to a unified output dimension .

[0047] Preprocessed psychological sandplay images Input the ResNet-50 backbone network, where The batch size is specified. Image features are extracted using convolutional layers, residual blocks, and global average pooling layers of ResNet-50 to obtain primary image features. Direct mapping via a simplified multilayer perceptron , .

[0048] From the original image Extracting color statistical features The intermediate image features are then processed using the FiLM fusion module. Adaptive modulation is performed by generating scaling factors using two independent linear transformations. With offset factor And calculate the final image features as ,in This indicates element-wise multiplication. This is used to constrain the range of modulation parameters, thereby explicitly injecting color prior information into the image representation.

[0049] For the first encoding layer of the operational feature encoder, the preprocessed primary operational features are... Input linear transformation layer for dimensionality compression ; right Batch normalization, ReLU activation, and Dropout processing are performed; for the second encoding layer of the operational feature encoder: The first layer output features Perform a linear transformation to map to a unified dimension. right Batch normalization, ReLU activation, and Dropout are performed to obtain the operational feature encoding output. .

[0050] The semantic feature encoder has the same structure as the operational feature encoder, and its input is the preprocessed sentiment semantic features. Through two layers of encoding processing, the semantic feature encoding output is obtained. .

[0051] The adaptive weight calculation module is used to calculate the adaptive weights of each modality in the current sample based on image features, operational features, and semantic features. The multimodal representation joint optimization module is used to generate a multimodal common representation based on the concatenated features of image features, operational features and semantic features, and to jointly optimize the multimodal common representation by using adaptive weights and combining intra-sample contrastive learning and inter-sample contrastive learning. The abnormal psychological sandplay classification module is used to input the optimized multimodal common representation into the classifier to determine whether the psychological sandplay of the target object is abnormal.

[0052] This application conducts experiments on large-scale real-world multimodal datasets MOSEI, MOSI, UR-FUNNY, and MUSTARD, and compares AMCF-SR with standard baseline methods, as detailed below: In the experimental setup, the mask ratio (masKratio) of the InfMasKing module was set to 0.5, and the number of masks was set to 5. The experimental results are shown in Table 1, where the linear classification task is evaluated based on top-1 accuracy (in%).

[0053] Table 1 Comparison Experiment of Public Datasets As shown in Table 1, AMCF-SR significantly outperforms Cross+Self, FactorCL, and CoMM on the MOSEI, UR-FUNNY, and MUSTARD datasets, validating the effectiveness of the adaptive weighting mechanism in the AMCF-SR method. On the MOSEI and MUSTARD datasets, AMCF-SR outperforms InfmasKing; their optimization strategies differ, with InfmasKing emphasizing the inclusion of collaborative information, while AMCF-SR focuses on modality weights and external consistency alignment of samples.

[0054] The inventors constructed a multimodal psychological sandplay dataset based on a digital psychological sandplay platform. The dataset includes sandplay images, operational procedures, and emotional semantic text related to the sandplay objects. The image input size is [size missing]. The dimensions of the operational process features and the emotional semantic features are 73 and 139, respectively. The overall dataset is divided into a training set of 4123 cases, a fine-tuning set of 1459 cases, and a test set of 626 cases, with the proportion of abnormal samples being less than 28%. To alleviate the class imbalance problem, the class weights were set to [1, 2.9] during the testing phase.

[0055] The model uses the AdamW optimizer, with an initial learning rate and a fine-tuning learning rate of 1.5%. A cosine annealing learning rate scheduling strategy with 10 rounds of warmup was used. The batch size was uniformly set to 64 for all experiments, with key hyperparameters including neighbor sampling number K=20, contrastive loss weight λ=0.5, and graph propagation hop count h=2. These configurations were initially optimized on the validation set and remained consistent across all comparative experiments to ensure fairness.

[0056] Table 2 Comparative Experiment Based on Sand Table Data Table 2 shows the anomaly detection performance of various methods on the mental sandplay dataset, based on F1-score. These include traditional supervised learning models (RandomFores, XGBoost, LightGBM), unsupervised methods (IsolationFores, LOF, COPOD), and deep learning methods (FactorCL, CoMM, InfMasKing). Table 2 shows that RandomFores has high precision but low recall, easily missing a low proportion of anomaly samples. XGBoost and LightGBM have relatively stable overall performance, but limited sensitivity to anomaly distribution, easily resulting in missed detections when the data distribution is imbalanced. Among the unsupervised methods, LOF and IsolationFores are affected by noise in local neighborhood estimation, leading to significant fluctuations in precision and recall. COPOD, relying on global distribution assumptions, struggles to fully characterize the complex anomaly patterns in the mental sandplay data. Among deep learning methods, FactorCL has a high recall but low precision, which may be related to error accumulation; CoMM has strong recall but slightly lower precision; InfMasKing has relatively balanced performance, but it mainly focuses on preserving information within samples and does not fully model global semantic relationships. In contrast, the AMCF-SR proposed in this paper focuses on global semantics and achieves the best results in both F1-score (0.847) and recall (0.857), demonstrating stronger anomaly detection capability and robustness.

[0057] The ablation experiments were conducted, and the results are shown in Tables 3 and 4. Table 3 shows the results of the module ablation experiments. SWCoMM refers to introducing an adaptive weighting mechanism on top of CoMM, while CoMMGraph refers to introducing a graph structure on top of CoMM to ensure external consistency alignment of samples. As can be seen from Table 3, SWCoMM with the adaptive weighting mechanism and CoMMGraph with graph structure modeling show slight improvements in AUC and F1-score compared to the basic CoMM model. The complete model AMCF-SR achieves the best results in key metrics such as AUC, F1-score, and precision, demonstrating the advantages of combining adaptive weighting mechanisms and external consistency alignment in balancing recognition accuracy and recall.

[0058] Table 3 Module Ablation Experiment Table The results of the modal ablation experiments are shown in Table 4. `image` represents the mental sandplay image, `emo` represents the emotional feature, and `ope` represents the operational feature. Modal ablation experiments were conducted based on AMCF-SR. The single-modal model showed limited performance in F1-score and AUC, with the operational feature (`ope`) performing the worst. The mental sandplay image (`image`) and emotional feature (`emo`) each captured some anomalous information. Multimodal fusion significantly improved the model's ability to identify anomalous samples. The bimodal combination generally outperformed the corresponding single-modal model, especially the combination including emotional features (`image+emo`, `emo+ope`), which performed better in F1-score and Recall. This indicates that the emotional modality can provide important supplementary information for anomalous mental sandplay identification, enhancing the model's sensitivity to anomalous samples. Furthermore, the trimodal joint modeling achieved the best performance in AUC, F1-score, and Precision-Recall balance, verifying the effectiveness and robustness of the multimodal fusion strategy for anomalous mental sandplay identification under imbalanced data conditions.

[0059] Table 4 Modal ablation experiment table To systematically evaluate the sensitivity of the AMCF-SR model to key hyperparameters, we conducted three sets of experiments on the mental sandplay dataset, targeting the number of neighbor samples K, the number of reachability hops h, and the redundancy suppression coefficient λ, to assess the model's sensitivity and robustness to key hyperparameters.

[0060] Table 5 shows the sensitivity analysis of the neighbor sampling number K under the conditions of fixed h=3 and λ=0.5. Experimental results show that when K increases from 5 to 20, the model's AUC and F1-score show an overall upward trend, reaching the optimum at K=20; after further increasing K to K=25, the performance slightly declines. This phenomenon indicates that moderately expanding the local neighborhood size helps the model to more fully aggregate structural information, thereby improving the stability and discriminativeness of the representation; however, an excessively large neighborhood size may introduce redundant or weakly correlated nodes, weakening the effectiveness of local structural constraints, thus interfering with the contrastive learning objective.

[0061] Table 5. The impact of different neighborhood size parameters K on model performance With k=20 and λ=0.5 fixed, the impact of the reachability hop count parameter h on model performance was analyzed, and the results are shown in Table 6. It can be observed that when h increases from 1 to 2, the model achieves significant improvements in both AUC and F1-score, while performance stabilizes at h=3 without significant further growth. This indicates that moderately expanding the structural receptive field helps the model establish a more discriminative global structural understanding. However, as the reachability continues to expand, the structural distinction between nodes gradually weakens, making it difficult for the model to obtain additional effective information from structural relationships at greater distances, thus leading to a saturation of performance gains.

[0062] Table 6 Comparison of model performance under varying parameter h values. Under the conditions of fixed h=2 and k=20, the redundancy suppression weight λ was analyzed, and the results are shown in Table 7. Experimental results show that as λ increases from 0.1 to 0.7, the model's F1-score and AUC continuously improve, reaching their optimum at λ=0.7; however, when λ further increases to 1.0, the performance declines. This trend indicates that appropriately enhancing the suppression of feature redundancy helps improve the discriminativeness of the representation, making the model more focused on sample uniqueness; however, excessive redundancy suppression weakens the necessary shared semantic structure between samples, leading to excessive dispersion of the representation space, thus affecting overall performance.

[0063] Table 7. Model performance evaluation results under different K settings In summary, the model achieves stable and superior performance across multiple key indicators with K=20, h=2, and λ=0.5~0.7, verifying its robustness to hyperparameter variations. This also demonstrates the importance of balancing local structure size, global receptive field, and redundancy suppression strength in improving the performance of abnormal psychological sandplay identification. In summary, this invention addresses the issue of single data modality by acquiring three modalities: sand table images, operational processes, and semantic text of sand table objects, thus providing a data foundation for multimodal fusion. Secondly, it introduces a self-weighting mechanism, calculating adaptive weights based on the information characteristics of each modality in the current sample. These weights dynamically adjust according to modal self-consistency and cross-modal geometric redundancy, resolving the problem of ignoring dynamic contribution differences between modalities. Finally, it employs a joint optimization strategy combining intra-sample contrastive learning and inter-sample contrastive learning. Intra-sample contrastive learning ensures consistency between different modalities within the same sample and maximizes the task relevance of high-quality modalities. Inter-sample contrastive learning mines semantic associations between samples by constructing a dynamic graph structure, thereby simultaneously capturing redundant, unique, and collaborative information between modalities. This invention can fully integrate complementary information from multimodal psychological sand table data, adaptively balance the contributions of each modality in the fusion process, and learn more discriminative and robust multimodal representations through dual consistency constraints both intra- and extra-sample. Therefore, in real-world scenarios with scarce labeled data and imbalanced samples, it significantly improves the accuracy and recall of identifying abnormal psychological sand tables.

[0064] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other occasions without modification, are all within the protection scope of the present invention.

Claims

1. A method for detecting abnormal psychological sandplay based on multimodal contrastive learning, characterized in that, Includes the following steps: Acquire multimodal data of psychological sandplay, including sandplay images, sandplay operation process, and semantic text of sandplay objects; Feature extraction is performed on the sand table image, the sand table operation process, and the semantic text of the sand table tools to obtain the corresponding image features, operation features, and semantic features. Data augmentation is performed on the sand table image, and the features extracted from the sand table operation process and the semantic text of the sand tools are enhanced to generate enhanced features corresponding to each modality; Based on the enhancement features of each modality, the self-consistency metric and cross-modal geometric redundancy metric of each modality are calculated. Based on the self-consistency metric and cross-modal geometric redundancy metric, a self-weighting mechanism is introduced to calculate the adaptive weight of each modality in the current sample. The image features, operational features, and semantic features are concatenated and a multimodal common representation is generated through a fusion network. The following two contrastive learning methods are then used to jointly optimize this multimodal common representation: Intra-sample contrastive learning: Align the enhanced features of different modalities of the same sample with their generated multimodal common representations, and align each single modal feature with its corresponding multimodal common representation. During the alignment process, the adaptive weights are used to weight the alignment strength of each modality. Inter-sample contrastive learning: Based on the similarity between the common representations of different samples in multimodal studies, an inter-sample relationship graph is constructed. Directly related sample pairs in the graph are used as positive samples, and unrelated sample pairs are used as negative samples. The common representations of positive sample pairs are aligned, while the common representations of negative sample pairs are separated. The optimized multimodal representation is input into a classifier to determine whether the target object's mental sandplay is an abnormal mental sandplay.

2. The abnormal psychological sandplay detection method based on multimodal contrastive learning according to claim 1, characterized in that, The calculation of the adaptive weights for each mode specifically includes: For each modality, the cosine similarity between its original features and the corresponding enhanced features is calculated as a measure of the modality’s self-consistency. For each modality, the average similarity between its original features and the original features of other modalities is calculated as a measure of cross-modal geometric redundancy. Based on the modal self-consistency and cross-modal geometric redundancy, a weight score is calculated for each mode, wherein the weight score is positively correlated with the modal self-consistency metric and negatively correlated with the cross-modal geometric redundancy metric. The weight scores of all modes are normalized to obtain the adaptive weights for each mode.

3. The abnormal psychological sandplay detection method based on multimodal contrastive learning according to claim 1, characterized in that, The inter-sample comparison learning includes: Construct a dynamic memory queue based on momentum updates to store the multimodal common representation of historical batch samples; Based on the multimodal common representation of the current batch of samples and all samples in the memory queue, the cosine similarity between samples is calculated, and the initial graph topology is constructed using the K-nearest neighbor sparsification strategy. On the K-nearest neighbor initial graph topology, sample pairs that are directly connected by an edge are used as positive sample pairs, and sample pairs that are unreachable in the multi-hop range are used as negative sample pairs. A graph contrastive learning loss function is constructed to optimize the multimodal common representation.

4. The abnormal psychological sandplay detection method based on multimodal contrastive learning according to claim 3, characterized in that, The initial graph topology of the K-nearest neighbor uses the mutual K-nearest neighbor algorithm to prune the initial directed edges, retaining only bidirectional connected edges to form an undirected graph adjacency matrix.

5. The abnormal psychological sandplay detection method based on multimodal contrastive learning according to claim 4, characterized in that, The classifier uses FocalLoss as the loss function during training.

6. The abnormal psychological sandplay detection method based on multimodal contrastive learning according to claim 1, characterized in that, The sand table operation process is feature extracted. Based on the operation time, operation type, operation object and spatial coordinate information in the process data, the emotional features in the operation process data are extracted from the time dimension, spatial dimension, operation dimension and sand table diversity dimension.

7. The abnormal psychological sandplay detection method based on multimodal contrastive learning according to claim 1, characterized in that, The semantic text of the sandplay objects is used for feature extraction, which includes constructing semantic features that reflect the overall positivity, emotional intensity, and thematic consistency of the sandplay based on a predefined sandplay object emotional symbol library.

8. An abnormal psychological sandplay detection system based on multimodal contrastive learning, based on the abnormal psychological sandplay detection method based on multimodal contrastive learning as described in any one of claims 1 to 7, characterized in that, include: The multimodal data acquisition module is used to acquire multimodal data of the target object's psychological sandplay, including sandplay images, sandplay operation process, and semantic text of sandplay objects; The multimodal feature extraction module includes an image encoder, an operation feature encoder, and a semantic feature encoder, which are used to extract and enhance features from the sand table image, the sand table operation process sequence, and the emotional semantic information of the sand table tools, respectively, to obtain corresponding image features, operation features, and semantic features. The adaptive weight calculation module calculates the self-consistency metric and cross-modal geometric redundancy metric for each modality based on the enhancement features of each modality. Based on the self-consistency metric and cross-modal geometric redundancy metric, a self-weighting mechanism is introduced to calculate the adaptive weight of each modality in the current sample. The multimodal representation joint optimization module is used to generate a multimodal common representation based on the concatenated features of the image features, operational features, and semantic features; and to jointly optimize the multimodal common representation by applying intra-sample contrastive learning and inter-sample contrastive learning; wherein, the intra-sample learning is used to align the multimodal common representations generated by different enhanced versions of the same sample, and to use the adaptive weights to align each single-modal feature with its corresponding multimodal common representation; the inter-sample contrastive learning is used to construct a relationship graph based on the similarity between samples, aligning directly related sample pairs in the graph and separating unrelated sample pairs; The abnormal psychological sandplay classification module is used to input the optimized multimodal common representation into the classifier to determine whether the psychological sandplay of the target object is an abnormal psychological sandplay.

9. The abnormal psychological sandplay detection system based on multimodal contrastive learning according to claim 8, characterized in that, The multimodal representation joint optimization module includes: The fusion unit is used to map the features of each modality to the same feature space as the common representation, and to identify the number of activated modalities with the help of the modality indicator vector, thereby generating the enhanced representation of each single modality and the global fusion representation; In-sample optimization unit, used to construct the in-sample contrastive learning loss function; The inter-sample optimization unit is used to construct the inter-sample contrastive learning loss function; The joint optimization unit is used to perform joint optimization of the multimodal common representation by weighting and combining the intra-sample contrast loss and the inter-sample contrast loss.

Images