A digital sand table multi-modal data analysis method based on an intelligent chat robot

Abstract

The present application relates to the technical field of artificial intelligence and psychology, and particularly relates to a digital sand table multi-modal data analysis method based on an intelligent chat robot, which comprises the following steps: constructing a knowledge-enhanced large language model architecture; using synonym replacement, scene generalization and dialogue rewriting strategy data enhancement on three types of training data sets; executing four-stage model training; synchronously collecting sand tool operation, facial expression, behavior and voice response data based on the large language model, synchronously integrating the multi-modal data sets according to timestamps, training a psychological analysis model; based on the explainability analysis of the screening results of the psychological analysis model, extracting the model attention weight and feature importance score, and performing attribution analysis on the sand table image, audio and video data to generate a text analysis report. The method can deeply mine subjective psychological motivation through dynamic interaction, realize deep fusion analysis of multi-modal data, and provide an explainability analysis report, and is suitable for depression risk screening of a large number of people.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of artificial intelligence and psychology, specifically to a method for multimodal data analysis of digital sandboxes based on intelligent chatbots. Background Technology

[0002] Digital sandplay, as an important tool in psychological counseling and therapy, is widely used in psychological assessment and intervention. Traditional digital sandplay systems mainly rely on the operator's selection and placement of sandplay objects for analysis, lacking dynamic interactive capabilities and making it difficult to deeply explore the operator's subjective psychological motivations. With the development of large language modeling technology, intelligent dialogue systems based on large language models have shown great potential in the field of psychological assessment. However, general-purpose large language models lack professional knowledge in the field of psychology, making it difficult to accurately understand the operation scenario of digital sandplay, and the generated questioning scripts lack professionalism and relevance.

[0003] In existing technologies, training methods for large language models are mainly geared towards general domains, lacking specialized training methods for digital sandbox psychological assessment scenarios. How to effectively integrate professional knowledge from the psychology domain into large language models, how to design targeted training processes and loss functions, and how to construct high-quality training datasets are all pressing technical problems that need to be solved. Furthermore, existing psychological assessment systems typically only output classification results, lacking interpretability analysis and failing to explain the contribution of each modality's data to the classification results. This makes it difficult for users to understand why a certain assessment result was obtained, limiting the system's credibility and practicality. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a multimodal data analysis method for digital sandboxes based on intelligent chatbots. By constructing a knowledge-enhanced large language model architecture and designing targeted training processes and loss functions, the large language model achieves professional training in digital sandbox psychological assessment scenarios, improving the professionalism, relevance, and effectiveness of the generated questioning scripts. Simultaneously, this invention also provides interpretability analysis capabilities. By extracting model attention weights and feature importance scores, attribution analysis is performed on sandbox image, audio, and video data, generating text analysis reports that explain the contribution of each modality of data to the classification results and their reasons, thereby enhancing the credibility and interpretability of the assessment results.

[0005] This invention is achieved through the following technical solution:

[0006] A method for multimodal data analysis of digital sandboxes based on intelligent chatbots is provided, including the following steps:

[0007] S1. Construct a knowledge-enhanced large language model architecture, including a basic language model layer, an external knowledge fusion layer, a prompting engineering module, and a scene adaptation layer;

[0008] S2. Construct training datasets, including a sandplay scenario-standard voice questioning script dataset, a clinical interview dialogue-typical questioning case dataset, and a sandplay scenario-response accuracy evaluation dataset, and perform data augmentation on the datasets;

[0009] S3. Perform knowledge-enhanced large language model training, including basic model loading and initialization, prompting engineering optimization training, incremental fine-tuning training, and model evaluation and optimization, using a hybrid loss function and optimization strategy;

[0010] S4. Based on the trained knowledge-enhanced large language model, synchronously collect sandplay data, facial expression data, behavioral data, and voice response data, integrate them into a multimodal dataset according to timestamps, and train a psychological analysis model for depression screening.

[0011] S5. Based on the depression screening results, perform interpretability analysis, extract model attention weights and feature importance scores, perform attribution analysis on multimodal data, and generate a text analysis report.

[0012] Furthermore, in step S1:

[0013] The basic language model layer adopts the Llama2-7B Chinese large model based on the Transformer architecture, which contains multiple Transformer blocks, each of which includes a multi-head self-attention mechanism and a feedforward neural network.

[0014] The external knowledge fusion layer includes a knowledge base interface module for the emotional attributes of sandplay objects and a clinical interview dialogue database interface module, which injects external knowledge through knowledge retrieval and vector fusion mechanisms.

[0015] The engineering module prompts the construction of composite prompt templates, combining system role definitions, the current sandplay scenario, knowledge base search results, and generation requirements.

[0016] The scene adaptation layer includes a scene recognition module, a script generation module, and a speech conversion module.

[0017] Furthermore, the knowledge retrieval and fusion process of the external knowledge fusion layer includes:

[0018] The knowledge base interface module for the emotional attributes of sandplay objects uses hash retrieval to return the energy attributes, emotional values, and symbolic meanings of the sandplay objects.

[0019] The clinical interview dialogue database interface module uses semantic similarity retrieval, based on Sentence-BERT encoding and cosine similarity calculation.

[0020] Knowledge fusion uses a learnable attention mechanism to weight and fuse the hidden state and knowledge vector of the model.

[0021] Furthermore, the data augmentation strategy in step S2 includes:

[0022] Synonym replacement: Randomly replace non-technical terms in the inquiry script with synonyms;

[0023] Scene generalization: Generate new operational scenes by changing the combination of sand objects;

[0024] Rewriting the script: Rewrite the sentences while keeping the semantics unchanged, ensuring that the semantic similarity threshold is not lower than 0.85.

[0025] Furthermore, in step S3:

[0026] The training process uses the AdamW optimizer, and the total loss function is a weighted combination of cross-entropy loss and semantic similarity loss;

[0027] A learning rate scheduling strategy is adopted, including cosine annealing and warm-up.

[0028] Employ gradient accumulation and gradient clipping strategies;

[0029] The model evaluation metrics include sandplay scenario-response accuracy and clinical dialogue matching, and the EarlyStopping mechanism is used.

[0030] Furthermore, in step S4, multimodal data acquisition includes:

[0031] Sandplay data includes timestamp, sandplay ID, energy attribute, and spatial coordinates.

[0032] Facial expression data: d-dimensional feature vectors;

[0033] Behavioral data: including hesitation duration, dwell time, and number of adjustments;

[0034] Voice response data includes speech rate, pitch, volume, keyword vectors, and sentiment vectors.

[0035] Multimodal datasets are synchronously integrated into a sequence of feature vectors according to timestamps.

[0036] Furthermore, in step S5, the interpretability analysis includes:

[0037] Attention weights and feature importance scores are extracted from the Transformer layer of the psychoanalytic model;

[0038] Attribution analysis was performed on the images, audio, and video data of the sand table to identify key features;

[0039] Generate a text report explaining the contribution of each modality of data to the classification results and the reasons therefor.

[0040] Furthermore, the composite prompt template structure of the prompting engineering module is: P = [System Role Definition] + [Current Sandplay Operation Scenario] + [Knowledge Base Retrieval Results] + [Generation Requirements]; the generation requirements include combining the symbolic meaning of the sandplay objects to guide, using empathetic open-ended questions, and avoiding sensitive questions.

[0041] Furthermore, the total loss function is calculated as follows:

[0042] ;

[0043] Among them: For cross-entropy loss, L sem The loss is for semantic similarity, where α and β are weight coefficients and α+β=1.

[0044] Furthermore, the psychoanalytic model employs a multimodal fusion architecture, including a multimodal encoder, a feature fusion layer, and a classifier, to output depression risk levels and key indicators.

[0045] The beneficial effects of this invention are:

[0046] I. Professional Training: By constructing a knowledge-enhanced large language model architecture, professional knowledge in the field of psychology is effectively integrated into the model, thereby improving the model's professionalism in digital sandplay psychological assessment scenarios.

[0047] II. Targeted Training Process: A four-stage training process was designed, including basic model loading, prompt engineering optimization, incremental fine-tuning, and model evaluation, to ensure that the model can generate professional and targeted questioning scripts.

[0048] III. Multiple Loss Function Design: A weighted combination of cross-entropy loss and semantic similarity loss is adopted to ensure both the fluency of the generated speech and the semantic consistency with the standard speech.

[0049] IV. Construction of High-Quality Datasets: Three types of training datasets were constructed, and various data augmentation strategies were adopted to improve the diversity and quality of the datasets.

[0050] V. Effective Evaluation Mechanism: A multi-dimensional evaluation index and an Early Stopping mechanism were designed to ensure the effectiveness and stability of model training.

[0051] VI. Interpretability Analysis: By extracting model attention weights and feature importance scores, a multimodal attribution analysis report is generated to explain the contribution of each modality of data to the classification results, thereby improving the credibility and interpretability of the results.

[0052] The method of this invention can accurately understand the operation scenario of digital sand table, generate professional and targeted questioning scripts, and provide detailed analysis reports through multimodal data fusion and interpretability analysis, effectively improving the accuracy, reliability and interpretability of digital sand table psychological assessment. Attached Figure Description

[0053] Figure 1 This describes the overall architecture and process of the present invention.

[0054] Figure 2 This is a schematic diagram of the multimodal data acquisition device in this invention. Detailed Implementation

[0055] To clearly illustrate the technical features of this solution, the following detailed implementation method will be used to explain the solution.

[0056] like Figure 1 As shown, a digital sandbox multimodal data analysis method based on intelligent chatbots includes five steps: constructing a knowledge-enhanced large language model architecture, constructing a training dataset, performing model training, multimodal data collection and psychological analysis model training, interpretability analysis, and report generation.

[0057] S1. Construct a knowledge-enhanced large language model architecture:

[0058] This invention adopts a knowledge-enhanced large language model based on the Transformer architecture. The overall architecture of the model includes four core components: a basic language model layer, an external knowledge fusion layer, a prompting engineering module, and a scene adaptation layer.

[0059] The base language model layer uses the Llama2-7B Chinese large-scale model as the pre-training foundation. Llama2-7B adopts a multi-layer Transformer encoder structure, containing N=32 Transformer blocks. Each Transformer block contains two sub-layers: a multi-head self-attention mechanism and a feedforward neural network.

[0060] The multi-head self-attention mechanism transforms the input sequence into a query matrix Q, a key matrix K, and a value matrix V through a linear transformation, calculates attention weights, and expands to h=32 heads. Each head calculates attention independently, and finally the results are concatenated and fused through the output projection matrix.

[0061] The feedforward neural network uses a two-layer fully connected network with a hidden dimension d. ff =11008, using the ReLU activation function.

[0062] The Transformer block employs layer normalization and residual connections, first performing multi-head self-attention computation, then performing feedforward neural network computation, with each layer containing residual connections and layer normalization.

[0063] The external knowledge fusion layer includes two knowledge base interface modules: a sandplay object emotional attribute knowledge base interface module and a clinical interview dialogue database interface module.

[0064] The knowledge base interface module for the emotional attributes of sand figurines uses a hash table for fast retrieval, with a retrieval time complexity of O(1).

[0065] ;

[0066] in: H represents the ID of the sand figurine, and H represents the hash table. The hash value of the sandplay object ID. Return the emotional attributes of the sandplay object, including its energy attributes. Emotional Value and symbolic meaning .

[0067] The clinical interview dialogue database interface module uses semantic similarity matching for retrieval:

[0068] Given the current sandplay scene C, semantic encoding is performed using Sentence-BERT:

[0069] ;

[0070] Calculate scene vectors Vectors of all clinical interview cases in the database Cosine similarity:

[0071] ;

[0072] Select the k=5 cases with the highest similarity as a reference:

[0073] ;

[0074] The knowledge fusion mechanism fuses the retrieved knowledge vectors with the model's hidden states:

[0075] First, the knowledge information is encoded into a vector representation:

[0076] ;

[0077] in For the retrieved knowledge text, For word embedding layer.

[0078] Then, the fusion weights are calculated using a learnable attention mechanism:

[0079] ;

[0080] in For the model's hidden state, This is the weight matrix. For bias terms, This represents vector concatenation. This is the activation function.

[0081] Finally, perform weighted fusion:

[0082] .

[0083] The tooltip module constructs a composite tooltip template P, which consists of four parts:

[0084] P = [System Role Definition] + [Current Sandplay Scenario] + [Knowledge Base Retrieval Results] + [Generation Requirements]

[0085] in:

[0086] [System Role Definition] = "You are a professional psychological assessor, skilled in guiding subjects to express their inner feelings through digital sandplay."

[0087] [Current Sandplay Scenario] = {Sandplay Operation Information: Sandplay Name} Energy Attributes Symbolic meaning };

[0088] [Knowledge Base Retrieval Results] = {Sandplay Emotional Attributes Knowledge: Emotional Value} Symbolic meaning Clinical Interview Reference Case: Relevant Questioning Techniques };

[0089] [Generation Requirements] = "1. Guide the conversation by incorporating the symbolic meaning of the sandplay objects; 2. Use empathetic and open-ended questioning techniques; 3. Avoid directly asking sensitive questions; 4. Use natural and fluent language suitable for voice interaction."

[0090] The composite prompt template P, filled with information about the sandplay scenario and knowledge base retrieval results, is input into the basic language model layer and encoded to obtain the prompt vector:

[0091] ;

[0092] Where L is the length of the cue sequence.

[0093] The scene adaptation layer consists of three modules:

[0094] (1) Scene recognition module: Identify the type of the current sandplay operation scene, including energy switching scene, negative sandplay ratio exceeding the threshold scene, etc.

[0095] (2) Script generation module: Generate targeted inquiry scripts based on scenario type and knowledge base retrieval results.

[0096] (3) Speech conversion module: converts the generated text speech into speech output using TTS (text-to-speech) technology.

[0097] S2. Construct the training dataset:

[0098] Construct three types of training datasets and perform data augmentation:

[0099] Dataset A: Sandplay Operation Scenarios - Standard Voice Interrogation Scripts Dataset;

[0100] We collected real-world digital sandplay simulation case studies, and several senior psychological assessors annotated standard questioning scripts for each key scenario. The data format is JSON and includes the following fields:

[0101] Unique identifier for the scene;

[0102] Trigger condition type (energy switching / negative percentage exceeding threshold);

[0103] : Context of sandplay operation (previous sandplay, current sandplay, energy switching direction, percentage of negative sandplay);

[0104] Standard inquiry script;

[0105] Knowledge base reference information;

[0106] Score for the quality of the sales pitch (1-5 points);

[0107] The dataset is divided into training, validation, and test sets in a ratio of 7:2:1.

[0108] Dataset B: Clinical Interview Dialogues - Typical Questioning Case Dataset;

[0109] Collaborate with the psychiatric departments of secondary and higher-level hospitals to collect depression screening interview records. Data preprocessing workflow:

[0110] Step 1: Speech-to-text conversion, using ASR (Automatic Speech Recognition) technology and the Whisper model for transcription.

[0111] Step 2: Desensitization process, removing sensitive information such as names and addresses, using regular expressions and named entity recognition technology.

[0112] Step 3: Key information annotation, with multiple psychiatrists annotating the question type (open-ended / closed-ended), level of empathy, and guidance strategies.

[0113] Step 4: Consistency verification. Perform Kappa consistency verification to ensure annotation quality. The Kappa coefficient must be ≥0.75.

[0114] The data is in JSON format and includes the following fields:

[0115] Unique identifier for the conversation;

[0116] Depression categories (mild / moderate / severe);

[0117] Dialogue rounds;

[0118] Clinical labels (empathy, guidance, assessment, etc.);

[0119] The dataset is divided into training, validation, and test sets in a ratio of 7:2:1.

[0120] Dataset C: Sandplay Operation Scenario - Response Accuracy Evaluation Dataset;

[0121] A simulated sandplay scenario was constructed, with a psychotester acting as the subject, responding to questions generated by a robot. The guiding effect of the questions was evaluated. Evaluation dimensions included: the relevance of the questions to the scenario, the accuracy of the use of professional terminology, the naturalness of empathetic expression, and the guiding effect. All data were used in the test set.

[0122] Data augmentation was performed on datasets A and B, including:

[0123] Synonym replacement: Replace non-technical terms in the standard inquiry script with synonyms. For each non-technical term in the Q script... From the thesaurus Randomly select a synonym replace:

[0124] ;

[0125] in Replacement probability These are the preset hyperparameters.

[0126] Scene generalization: Based on existing scenes, new operational scenes are generated by varying the combination of sand objects. (For the scene...) New scenes are generated by changing the combination of sand objects. :

[0127] ;

[0128] in It is a sandplay tool library, containing 1000+ types of sandplay tools. This indicates a uniformly distributed sampling.

[0129] Rewriting the script: Rewrite the sentence structure of the inquiry while maintaining the original meaning. Generate semantically equivalent new terms through grammatical transformation The rewritten wording must meet the following requirements:

[0130] ;

[0131] in The semantic similarity function is calculated using Sentence-BERT:

[0132] ;

[0133] This is a preset semantic similarity threshold.

[0134] S3, Perform model training:

[0135] Model training is divided into four stages:

[0136] Phase 1: Basic Model Loading and Initialization

[0137] From Load the pre-trained weight parameters of the Llama2-7B Chinese large-scale model from the model library. The model has a total of N=32 layers. Before freezing, there were 24 layers with NL parameters. Only the last 8 layers (L=8) were fine-tuned.

[0138] ;

[0139] ;

[0140] in For the set of trainable parameters, This refers to freezing the parameter set. Freezing the underlying parameters can reduce training costs, prevent overfitting, and preserve the general language understanding capabilities of the pre-trained model.

[0141] Initialize the knowledge base interface:

[0142] (1) Load the "Sand Figure Emotional Attribute Knowledge Base" (CSV format) and construct a hash index:

[0143] ;

[0144] in For sand figurine ID, It is an energy attribute. For sentiment value, It has symbolic meaning.

[0145] (2) Load the Clinical Interview Dialogue Database (JSON format) and construct a semantic vector index:

[0146] ;

[0147] in For the i-th clinical interview case, The total number of cases, This is the semantic vector representation of the case.

[0148] Phase Two: Training on Engineering Optimization

[0149] Based on datasets A and B, we analyze frequently occurring question patterns and extract common features. We then construct a composite prompt template that includes "system role - scene information - knowledge base content - generation requirements".

[0150] The structure and content of the prompt templates were optimized through manual evaluation and A / B testing. Validation sets (the validation portions of datasets A and B) were used to test the generation effectiveness of different prompt templates, with evaluation metrics including: question relevance to the scenario and accuracy of terminology usage.

[0151] Select the best-performing prompt template as the fixed template and keep it unchanged in subsequent training.

[0152] Phase 3: Incremental Fine-tuning Training

[0153] Datasets A and B were integrated into a unified training sample format. Each training sample contained: input a composite prompt template (filled with information about the sandplay scenario and knowledge base retrieval results); output a standard question script (from dataset A) or clinical question content (from dataset B).

[0154] Training parameter settings:

[0155] Optimizer: AdamW optimizer;

[0156] Initial learning rate: ;

[0157] Minimum learning rate: ;

[0158] Batch size: ;

[0159] Gradient accumulation steps: (The effective batch size is 32);

[0160] Maximum number of training rounds: ;

[0161] Weight decay coefficient: ;

[0162] Gradient clipping threshold: ;

[0163] Warm-up rounds: step;

[0164] Learning rate scheduling strategy:

[0165] A cosine annealing scheduling strategy is adopted, and the learning rate increases from its maximum value with the number of training steps. Gradually decay to minimum value .

[0166] Employing a warm-up strategy, before The learning rate is increased linearly during step training.

[0167] Loss function design:

[0168] Total loss function It consists of cross-entropy loss and semantic similarity loss:

[0169] ;

[0170] in and These are the weight coefficients for cross-entropy loss and semantic similarity loss, respectively. .

[0171] Cross-entropy loss The probability distribution used to predict the next token:

[0172] ;

[0173] Where N is the sequence length, and |V|=32000 is the vocabulary size. One-hot encoding for the real label, The probability distribution predicted by the model:

[0174] ;

[0175] in Let i be the hidden state at position i. To output the projection matrix, This is a bias term.

[0176] The softmax function is calculated as follows:

[0177] ;

[0178] Semantic similarity loss Used to ensure semantic consistency between generated scripts and standard scripts:

[0179] ;

[0180] in Cosine similarity function:

[0181] ;

[0182] and The vector representations of the model-generated and standard question phrases are respectively calculated using Sentence-BERT:

[0183] ;

[0184] ;

[0185] Training process:

[0186] 1. Forward Propagation: The training samples are input into the model, passing through the basic language model layer, the external knowledge fusion layer, and the prompting engineering module to generate the question script. .

[0187] 2. Calculate the loss value between the generated script and the standard script:

[0188] ;

[0189] 3. Backpropagation: Calculate the gradient, only for trainable parameters.

[0190] ;

[0191] ▽ represents the gradient operator, which indicates the gradient of a function, i.e., the vector of all partial derivatives; θ trainable The set of trainable parameters includes parameters of the external knowledge fusion layer, parameters of the prompting engineering module, and parameters of the L-layer of the Llama2-7B model. These parameters are optimized and adjusted during training based on the sand table placement query utterance generation task and clinical dialogue data. L total The total loss function represents the overall loss value of the knowledge-enhanced large language model in the task of generating query discourses in a sandbox setting in this invention. This is expressed as taking the partial derivative with respect to that variable.

[0192] The gradient of the freeze parameter is 0:

[0193] ;

[0194] θ frozen In this invention, the term "frozen parameter set" specifically refers to parameters that are not updated during the training of the knowledge-enhanced large language model. This includes the basic parameters of the first NL layers of the Llama2-7B pre-trained model. These parameters remain fixed during the fine-tuning training phase to preserve the general language knowledge and comprehension abilities learned by the pre-trained model. L total The total loss function is defined by the formula, which indicates that the gradient of the total loss function with frozen parameters is zero. This means that during backpropagation, the base layer parameters of the Llama2-7B pre-trained model do not participate in gradient calculation and parameter updates, keeping their pre-trained weights unchanged. This allows the model to maintain its general capabilities while adapting to professional scenarios of digital sandbox psychological assessment by updating trainable parameters.

[0195] 4. Gradient processing:

[0196] A gradient accumulation strategy is adopted, in which the parameters are updated once after accumulating gradients every G=4 batches, thereby increasing the effective batch size.

[0197] Gradient clipping is used to prevent gradient explosion. Clipping is performed when the L2 norm of the gradient vector exceeds a preset threshold of 1.0.

[0198] 5. Parameter update: The AdamW optimizer is used to update the parameters. The first and second moments of the gradient are used for adaptive learning rate updates, and weight decay is used to prevent overfitting.

[0199] 6. Validation and Evaluation: After each training round, evaluate the model performance on the validation set and calculate the evaluation metrics.

[0200] Phase Four: Model Evaluation and Optimization

[0201] Evaluation index calculation:

[0202] Indicator 1: Sandplay Operation Scenario - Response Accuracy Rate;

[0203] Using dataset C, the model generates questions, which are then independently evaluated by multiple psychotesters, and the average score is taken. Evaluation dimensions include scenario relevance. Accuracy of professional terminology and empathy Each of these is calculated as a percentage of the total number of inquiries for the corresponding indicator.

[0204] Overall accuracy (Acc) is a weighted combination:

[0205] ;

[0206] in The weighting coefficients and ;

[0207] Scene matching The calculation is as follows:

[0208] ;

[0209] Accuracy of technical terms The calculation is as follows:

[0210] ;

[0211] Empathy Naturalness The calculation is as follows:

[0212] .

[0213] Indicator 2: Clinical Script Matching Sim:

[0214] Sim uses Sentence-BERT to calculate cosine similarity for clinical dialogue matching.

[0215] ;

[0216] Where M is the number of test samples, and These are the generated and standard scripts for the i-th sample, respectively.

[0217] The Early Stopping mechanism monitors the overall accuracy on the validation set, and the stopping condition is:

[0218] After E rounds of training, the overall accuracy on the validation set did not improve.

[0219] ;

[0220] in Let E be the accuracy of the validation set in round t, and E be the early stopping patience value.

[0221] Using clinical interview data from the test set, the mean cosine similarity between the generated statements and standard clinical statements was calculated using the Sentence-BERT computational model.

[0222] Early Stopping mechanism: The monitoring metric is the overall accuracy on the validation set. If the overall accuracy of the validation set does not improve after E=3 consecutive training rounds, then training should be stopped. E=3 is the early stopping patience value.

[0223] Model saving: Save the optimal model parameters on the validation set and record the corresponding evaluation index values.

[0224] Model performance validation: Evaluate the final model performance on the test set to ensure it meets the target metrics (Acc≥85%, Sim≥0.80). Generate sample question scripts, which are then manually evaluated by multiple senior psychological assessors to ensure that the questions generated by the model conform to clinical psychological assessment standards.

[0225] If the target metrics are not met, analyze the failure cases, adjust the training strategy (such as increasing training data, adjusting the weights α and β of the loss function, optimizing the prompt template, etc.), and retrain.

[0226] S4. Data collection and psychological analysis model training based on knowledge-enhanced large language models:

[0227] When the operator operates the digital sand table, multimodal data is collected simultaneously, such as... Figure 2 The multimodal data acquisition device shown.

[0228] Data collection of sandplay operation ,in t is the timestamp, si is the sand tool ID, ei is the energy attribute, and xi and yi are the spatial coordinates;

[0229] Collect facial expression data ,in Let d represent the d-dimensional facial feature vector at time t;

[0230] Collect behavioral data ,in , For the duration of hesitation, Duration of stay To adjust the number of times;

[0231] Collect voice response feature data ,in v represents speech rate, p represents pitch, l represents volume, kw represents keyword vector, and em represents sentiment vector;

[0232] Integrate into a multimodal dataset based on timestamp synchronization principles. ,in Let represent the multimodal eigenvector at time t.

[0233] Based on the collected multimodal dataset M and corresponding depression screening labels, a training dataset for the psychoanalysis model was constructed. The psychoanalysis model employs a multimodal fusion architecture, including a multimodal encoder, a feature fusion layer, and a classifier. The multimodal encoder encodes the data from sandplay manipulation, facial expressions, behavior, and speech responses, respectively, to obtain feature representations for each modality. The feature fusion layer fuses the features from each modality to obtain a fused multimodal feature vector. The classifier classifies depression risk levels based on the fused multimodal feature vector, outputting the depression risk level, risk probability, and key risk correlation indicators. The model is trained using a cross-entropy loss function and the AdamW optimizer, and the optimal model is saved using an Early Stopping mechanism.

[0234] S5. Interpretability Analysis and Report Generation:

[0235] Based on the depression screening results obtained in step S4, a multimodal attribution analysis report is generated using the interpretability analysis module:

[0236] Extracting attention weights from intermediate layers of the model: Extracting the attention weight matrix from the Transformer layer of the psychoanalysis model trained in step S4 to identify the degree of attention the model pays to data of different modalities.

[0237] Calculate feature importance scores: Based on the gradient information output by the model, calculate the contribution of each modality feature to the final classification result, including the importance of sand table image features, audio features, and video features.

[0238] Multimodal attribution analysis: Attribution analysis is performed on sand table images, audio features, and video features based on attention weights and feature importance scores to identify key features.

[0239] Generate a text analysis report: Based on the attribution analysis results, a natural language text report is generated using a template-based method. The report includes: an overview of the classification results, a ranking of the contributions of each modality to the results, key feature identification, and an explanation of the attribution reasons.

[0240] To verify the effectiveness of the proposed model, this invention was evaluated on three publicly available large models (GPT-3.5-turbo, GPT-4, and Llama2-7B fine-tuning). The performance comparison data are shown in Tables 1 and 2 below. Through comparison and observation, it can be seen that the method of this invention has achieved significant improvements in both core tasks.

[0241] Table 1: Performance Comparison of Query Phrase Generation Tasks

[0242] Model Accuracy Precision Recall F1 score Clinical conversation matching degree (Sim) GPT-3.5-turbo 71.5% 69.2% 73.8% 0.714 0.678 GPT-4 75.8% 73.6% 78.1% 0.758 0.712 Llama2-7B Fine-tuning 78.9% 76.5% 81.3% 0.788 0.745 Knowledge-enhanced large language model 88.5% 86.2% 90.7% 0.883 0.830

[0243] In Table 1, the accuracy of the method of this invention reached 88.5% and the F1 score was 0.883 in the query utterance generation task. This represents a 9.6% improvement compared to the best comparison method (Llama2-7B fine-tuning). This indicates that through external knowledge fusion and prompting engineering optimization, the model can generate more professional, natural and clinically compliant query utterances.

[0244] Table 2: Performance Comparison of Psychological Risk Screening Tasks

[0245] Model Accuracy Precision Recall F1 score Clinical conversation matching degree (Sim) GPT-3.5-turbo 72.8% 70.5% 75.2% 0.650 0.682 GPT-4 74.2% 78.9% 78.1% 0.765 0.789 Llama2-7B Fine-tuning 79.6% 77.3% 82.1% 0.796 0.823 Psychoanalytic model 83.2% 81.5% 84.8% 0.831 0.865

[0246] In Table 2, the method of this invention achieved an accuracy of 83.2% and an F1 score of 0.831 in the psychological risk screening task, with a clinical dialogue matching degree of 0.865. This represents a 3.6% improvement compared to the best comparison method, indicating that through multimodal data fusion and knowledge enhancement, the model exhibits higher accuracy and reliability in psychological risk identification. Overall, through external knowledge fusion and multimodal data fusion, the method of this invention effectively improves the accuracy and reliability of the model in professional psychological assessment scenarios, providing a more precise analytical tool for digital sandplay psychotherapy.

[0247] Application examples:

[0248] This invention was applied to the psychological counseling center of a university to conduct a psychological evaluation of a college student (pseudonym Xiao Li) who was suspected of having depressive tendencies. Xiao Li had recently exhibited behaviors such as low mood and social avoidance.

[0249] Implementation process:

[0250] 1. Initial Sandbox Operation and Dynamic Interaction:

[0251] Xiao Li first selected a sandplay object representing "loneliness" (sandplay object ID: s123, energy attribute eᵢ= -1, emotional value eᵥ = -1) on the digital sandplay and placed it in the corner of the sandplay.

[0252] The knowledge-enhanced large language model, through its scene recognition module, identifies this as a "negative initial choice of a figurine." The model then retrieves the emotional attributes (symbolic meaning) of the figurine using an external knowledge fusion layer. =“isolated, helpless”) and similar clinical interview cases.

[0253] The prompting module generates a composite prompt template, the model generates the first round of questioning script, and the output is through the speech conversion module: "I noticed that you chose a somewhat lonely sand figurine and placed it in the corner. Would you like to talk more about how you feel right now?"

[0254] 2. Multimodal data acquisition:

[0255] In subsequent interactions, the system synchronously collects the following multimodal data (timestamp t) according to timestamps:

[0256] Sandplay operation data ( o Xiao Li is hesitating. = 5.2 seconds later, a new sand item representing "binding" (s456, eᵢ = -1) was added and placed next to the first sand item.

[0257] Facial expression data ( The camera captured feature vectors of a sad expression, characterized by downturned corners of the mouth and raised eyebrows. .

[0258] Behavioral data ( b t ): Recorded multiple sandplay adjustments (n) h = 3 times) and long stay (τ) s = 15 seconds).

[0259] Voice response data (v t When Xiao Li answered the model's questions, his speech features showed a slow speaking speed (v = 3.2 words / second), a low tone (p = 85 Hz), and contained the keywords "stress" and "can't sleep" (highlighted by the kw vector).

[0260] 3. Screening and interpretability analysis of psychoanalytic models

[0261] All multimodal data are integrated into a sequence M = {m1, m2, ..., m} by timestamp. T}, input into the trained psychoanalysis model.

[0262] After comprehensive analysis, the model outputs a depression risk level of "moderate risk" with a probability of 72%.

[0263] Interpretability analysis module started:

[0264] Feature importance analysis: Calculations showed that the sandplay manipulation pattern (negative sandplay cluster) contributed the most (35%), followed by the phonological keyword "can't sleep" (25%) and facial sadness expression (20%).

[0265] Attention weight analysis: The attention weights extracted from the Transformer layer of the model show that the model has the highest attention peak when processing the "binding" sand figure (s456) and voice keywords.

[0266] Attribution analysis revealed that "clustering of negative sandplay objects" was a key spatial feature, while "slow speech rate and the appearance of keywords" were key audio features.

[0267] 4. Generate a text analysis report

[0268] The system automatically generates the following natural language report:

[0269] Screening results: Moderate risk of depression (72% probability).

[0270] Main basis:

[0271] Sandplay analysis (contribution 35%): Subjects consecutively selected two sandplay objects with high negative affective values ​​("loneliness and confinement") and arranged them in a close clustering manner. This pattern is often associated with feelings of helplessness and self-isolation tendencies in clinical practice.

[0272] Voice feature analysis (contribution 25%): The subjects' speech rate was significantly lower than the average level, and they explicitly mentioned keywords such as "can't sleep", which is consistent with the speech characteristics of depressive symptoms.

[0273] Facial expression and behavior analysis (contribution 30%): The patient consistently displayed a sad expression and hesitated and made adjustments frequently during the operation, reflecting decision-making difficulties and a depressed mood.

[0274] Recommendation: Follow-up clinical interviews are recommended to further confirm the evaluation results.

[0275] Of course, the above description is not limited to the examples above. Technical features not described in this invention can be implemented by or using existing technology, and will not be repeated here. The above embodiments and drawings are only used to illustrate the technical solutions of this invention and are not intended to limit this invention. This invention has been described in detail with reference to preferred embodiments. Those skilled in the art should understand that any changes, modifications, additions or substitutions made by those skilled in the art within the scope of this invention do not depart from the spirit of this invention and should also fall within the scope of protection of the claims of this invention.

Claims

1. A method for multimodal data analysis of a digital sandbox based on an intelligent chatbot, characterized in that: Includes the following steps: S1. Construct a knowledge-enhanced large language model architecture, including a basic language model layer, an external knowledge fusion layer, a prompting engineering module, and a scene adaptation layer; the basic language model layer adopts the Llama2-7B Chinese large model based on the Transformer architecture, which contains multiple Transformer blocks, each of which includes a multi-head self-attention mechanism and a feedforward neural network. The external knowledge fusion layer includes a knowledge base interface module for the emotional attributes of the sandplay tool and a clinical interview dialogue database interface module, which inject external knowledge through knowledge retrieval and vector fusion mechanisms; the prompting engineering module constructs composite prompt templates, combining system role definitions, the current sandplay tool operation scenario, knowledge base retrieval results, and generation requirements; the scenario adaptation layer includes a scenario recognition module, a script generation module, and a speech conversion module. S2. Construct training datasets, including a sandplay scenario-standard voice questioning script dataset, a clinical interview dialogue-typical questioning case dataset, and a sandplay scenario-response accuracy evaluation dataset, and perform data augmentation on the datasets; S3. Perform knowledge-enhanced large language model training, including basic model loading and initialization, prompting engineering optimization training, incremental fine-tuning training, and model evaluation and optimization, using a hybrid loss function and optimization strategy; S4. Based on the trained knowledge-enhanced large language model, synchronously collect sandplay data, facial expression data, behavioral data, and voice response data, integrate them into a multimodal dataset according to timestamps, and train a psychological analysis model for depression screening. Multimodal data acquisition includes: Sandplay data: including timestamps, sandplay IDs, energy attributes, and spatial coordinates; facial expression data: d-dimensional feature vectors; behavior data. Data includes hesitation duration, dwell time, and number of adjustments; voice response data includes speech rate, pitch, volume, keyword vectors, and sentiment vectors; the multimodal dataset is synchronously integrated into a feature vector sequence according to timestamps; S5. Based on the depression screening results, perform interpretability analysis, extract model attention weights and feature importance scores, perform attribution analysis on multimodal data, and generate a text analysis report.

2. The method for multimodal data analysis of digital sandbox based on intelligent chatbot according to claim 1, characterized in that: The knowledge retrieval and fusion process of the external knowledge fusion layer includes: The knowledge base interface module for the emotional attributes of sandplay objects uses hash retrieval to return the energy attributes, emotional values, and symbolic meanings of the sandplay objects. The clinical interview dialogue database interface module uses semantic similarity retrieval, based on Sentence-BERT encoding and cosine similarity calculation. Knowledge fusion uses a learnable attention mechanism to weight and fuse the hidden state and knowledge vector of the model.

3. The method for multimodal data analysis of digital sandbox based on intelligent chatbots according to claim 1, characterized in that: The data augmentation strategies in step S2 include: Synonym replacement: Randomly replace non-technical terms in the inquiry script with synonyms; Scene generalization: Generate new operational scenes by changing the combination of sand objects; Rewriting the script: Rewrite the sentences while keeping the semantics unchanged, ensuring that the semantic similarity threshold is not lower than 0.

85.

4. The method for multimodal data analysis of digital sandbox based on intelligent chatbot according to claim 1, characterized in that: In step S3: The training process uses the AdamW optimizer, and the total loss function is a weighted combination of cross-entropy loss and semantic similarity loss; A learning rate scheduling strategy is adopted, including cosine annealing and warm-up. Employ gradient accumulation and gradient clipping strategies; The model evaluation metrics include sandplay scenario-response accuracy and clinical dialogue matching, and the EarlyStopping mechanism is used.

5. The method for multimodal data analysis of digital sandbox based on intelligent chatbot according to claim 1, characterized in that: In step S5, the interpretability analysis includes: Attention weights and feature importance scores are extracted from the Transformer layer of the psychoanalytic model; Attribution analysis was performed on the images, audio, and video data of the sand table to identify key features; Generate a text report explaining the contribution of each modality of data to the classification results and the reasons therefor.

6. The method for multimodal data analysis of digital sandbox based on intelligent chatbot according to claim 2, characterized in that: The composite prompt template structure of the prompt engineering module is: P=[System Role Definition]+[Current Sandplay Operation Scenario]+[Knowledge Base Retrieval Results]+[Generation Requirements]; the generation requirements include guiding based on the symbolic meaning of the sandplay, using empathetic open-ended questions, and avoiding sensitive questions.

7. The method for multimodal data analysis of digital sandboxes based on intelligent chatbots according to claim 4, characterized in that: The total loss function is calculated as follows: ； in: For cross-entropy loss, L sem For semantic similarity loss, α and β are weight coefficients and α+β=1.

8. The method for multimodal data analysis of digital sandbox based on intelligent chatbot according to claim 1, characterized in that: The psychoanalysis model employs a multimodal fusion architecture, including a multimodal encoder, a feature fusion layer, and a classifier, to output depression risk levels and key indicators.

Images