Interview response psychological twinborn modeling method based on multi-modal data fusion

Abstract

The invention relates to an interview response psychological twinborn modeling method based on multi-modal data fusion, and relates to the technical field of twinborn modeling, and the method comprises the steps: determining a plurality of key test scenes, and obtaining multi-modal performance test data of a plurality of test users under the plurality of key test scenes; determining a plurality of user types based on the multi-modal performance test data of the plurality of test users in the plurality of key test scenes; obtaining multi-modal interview response data of a plurality of test users; constructing a psychological expression knowledge graph based on the multi-modal interview response data of the plurality of test users; obtaining multi-modal performance test data of an interviewer in a plurality of key test scenes, and determining a user type to which the interviewer belongs; obtaining multi-modal interview response data of the interviewer; based on the multi-modal interview response data of the interviewer, the user type to which the interviewer belongs and the psychological expression knowledge map, response psychological twinborn modeling of the interviewer is carried out, and the method has the advantage of improving the accuracy of interview response psychological assessment.

Description

Technical Field

[0001] This invention relates to the field of twin modeling technology, and in particular to a method for psychological twin modeling of interview responses based on multimodal data fusion. Background Technology

[0002] Interview response psychology refers to the immediate psychological reactions and behavioral performances of job applicants during an interview, triggered by external stimuli (such as interviewer questions and environmental pressure). It encompasses multiple dimensions, including emotional state (such as nervousness and confidence), cognitive processes (such as information processing and logical expression), and behavioral tendencies (such as body language and vocal characteristics). In recruitment scenarios, accurate assessment of interview response psychology is a crucial basis for talent selection. Research shows that applicants' nonverbal behaviors (such as body language openness) are highly correlated with soft skills such as teamwork and resilience. Traditional assessment methods often overlook this information, leading to a 30%-40% discrepancy between assessment results and actual work performance. Therefore, developing a technological solution that can comprehensively capture the dynamic characteristics of interview response psychology is of significant value in improving recruitment efficiency and talent matching.

[0003] Traditional psychological assessments during interviews primarily rely on structured interviews, psychological scales, or single-modal data analysis (such as judging emotions solely through voice tone). These methods have the following drawbacks: Limited data dimensions: Relying solely on text or voice information fails to capture nonverbal cues such as micro-expressions and body language. For example, job applicants may use language to mask their true emotions, but micro-expressions (such as a brief frown) or body language (such as crossing arms) can reveal their underlying psychological state.

[0004] Static analysis is the primary method: Traditional methods are mostly based on offline data or static scores, making it difficult to reflect the dynamic changes in the candidate's psychological state during the interview process in real time. For example, when answering stressful questions, a candidate may shift from confidence to anxiety, but traditional methods cannot capture this transition.

[0005] Subjective interference: The interviewer's subjective judgment may introduce bias, resulting in a lack of objectivity in the evaluation results.

[0006] Therefore, there is a need to provide a method for psychological twin modeling of interview responses based on multimodal data fusion, in order to improve the accuracy of psychological assessment of interview responses. Summary of the Invention

[0007] This invention provides a method for psychological twin modeling of interview responses based on multimodal data fusion, comprising: identifying multiple key test scenarios and acquiring multimodal performance test data of multiple test users in multiple key test scenarios; identifying multiple user types based on the multimodal performance test data of multiple test users in multiple key test scenarios; acquiring multimodal interview response data of multiple test users; constructing a psychological performance knowledge graph based on the multimodal interview response data of multiple test users, wherein the psychological performance knowledge graph is used to record multiple key psychological performance factors of different user types; acquiring multimodal performance test data of interviewees in multiple key test scenarios and determining the user type to which the interviewee belongs; acquiring multimodal interview response data of interviewees; and performing psychological twin modeling of interviewees' responses based on the multimodal interview response data of interviewees, the user type to which the interviewee belongs, and the psychological performance knowledge graph.

[0008] Furthermore, several key test scenarios are identified, including: determining a multimodal psychological performance factor set and multiple test scenarios, wherein the multimodal psychological performance factor set includes psychological performance factors corresponding to multiple modalities; acquiring multimodal performance test data of multiple test users in multiple test scenarios; for each test user and each test scenario, constructing a multimodal response feature matrix for the test user corresponding to the test scenario based on the test user's multimodal performance test data and the multimodal psychological performance factor set, wherein one row vector in the multimodal response feature matrix corresponds to one modality; for each test scenario, calculating the psychological performance difference value corresponding to the test scenario based on the multimodal response feature matrix for each test user corresponding to the test scenario; determining candidate test scenarios based on the psychological performance difference value corresponding to each test scenario; for any two candidate test scenarios, calculating the psychological performance similarity between the two candidate test scenarios based on the multimodal response feature matrices for each test user corresponding to the two candidate test scenarios; and determining multiple key test scenarios based on the psychological performance similarity between any two candidate test scenarios.

[0009] Furthermore, multimodal performance test data of test users in key test scenarios are obtained, including: visual modal response test data of test users in key test scenarios through image acquisition devices; auditory modal response test data and language modal response test data of test users in key test scenarios through voice acquisition devices; and behavioral modal response test data of test users in key test scenarios through pressure acquisition devices installed on the seats.

[0010] Furthermore, the pressure acquisition device includes multiple pressure sensing devices installed at different positions on the seat; the installation positions of the multiple pressure sensing devices are determined based on the following process: identifying multiple psychological induced scenarios; for each psychological induced scenario, acquiring seat pressure data of multiple test users in the psychological induced scenario, wherein the seat pressure data includes pressure sequences at multiple positions on the seat; and determining the installation positions of the multiple pressure sensing devices based on the seat pressure data of multiple test users in the psychological induced scenario using an improved butterfly optimization algorithm.

[0011] Furthermore, in the improved butterfly optimization algorithm, the fitness of the butterfly is determined based on the differentiated weights of the psychologically induced scenario, and the switching probability is determined based on the similarity of the current global optimal solution during iteration.

[0012] Furthermore, based on multimodal performance test data from multiple test users across multiple key test scenarios, multiple user types are identified, including: for each test user and each key test scenario, constructing a key multimodal response feature matrix for the test user corresponding to the key test scenario based on the test user's multimodal performance test data in the key test scenario; for any two test users, calculating the clustering distance between the two test users based on the key multimodal response feature matrices for each key test scenario; and using a clustering algorithm, clustering multiple test users according to the clustering distance between any two test users to determine multiple user types.

[0013] Furthermore, the multimodal interview response data of the test users includes visual modal interview response data, auditory modal interview response data, verbal modal interview response data, and behavioral modal interview response data. Based on the multimodal interview response data of multiple test users, a psychological performance knowledge graph is constructed, including: for each user type, based on the key multimodal response feature matrix of the test users corresponding to each key test scenario included in the user type, calculating the factor performance difference value of each psychological performance factor corresponding to the user type; based on the factor performance difference value of each psychological performance factor corresponding to the user type, determining multiple key psychological performance factors of the user type; and constructing a psychological performance knowledge graph based on the multiple key psychological performance factors of each user type.

[0014] Furthermore, based on the key multimodal response feature matrix of the test users corresponding to each key test scenario, including the user types, the factor performance difference value of each psychological performance factor corresponding to the user type is calculated, including: for any two key test scenarios, based on the key multimodal response feature matrix of the test users corresponding to the two key test scenarios, the factor performance difference value of the psychological performance factor corresponding to the two key test scenarios is calculated; based on the factor performance difference value of the psychological performance factor corresponding to any two key test scenarios, the factor performance difference value of the psychological performance factor corresponding to the user type is calculated.

[0015] Furthermore, the user type of the interviewee is determined, including: constructing a key multimodal response feature matrix for each key test scenario based on the interviewee's multimodal performance test data in multiple key test scenarios; for each user type, constructing a key multimodal response feature matrix for each key test scenario based on the key multimodal response feature matrices of the test users included in the user type; and determining the user type of the interviewee based on the key multimodal response feature matrix for each key test scenario and the key multimodal response feature matrix for each user type.

[0016] Furthermore, based on the interviewee's multimodal interview response data, the interviewee's user type, and the psychological performance knowledge graph, a psychological twin model of the interviewee's response is constructed. This includes: for each user type, constructing a psychological twin model of the response corresponding to the user type based on the psychological performance knowledge graph; retrieving the psychological twin model of the response corresponding to the interviewee's user type, and constructing a psychological twin model of the interviewee's response based on the interviewee's multimodal interview response data.

[0017] Compared to existing technologies, the interview response psychological twin modeling method based on multimodal data fusion provided in this specification has at least the following beneficial effects: 1. By fusing multimodal data, a psychological performance feature matrix is ​​constructed. Combined with dynamically selected key test scenarios, this breaks through the limitations of traditional single-modal or fixed-scenario assessments.

[0018] 2. Based on the knowledge graph of user types and psychological performance extracted from test user data, it can quantify the core psychological characteristics of different user types and eliminate subjective judgment bias; 3. By dynamically determining the layout of seat pressure sensing devices through an improved butterfly optimization algorithm, and combining it with differentiated weights for psychologically induced scenarios, we can achieve more accurate and cost-effective behavioral data collection, avoid redundant collection, and improve data effectiveness. Attached Figure Description

[0019] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein: Figure 1 This is a flowchart illustrating a method for psychological twin modeling of interview responses based on multimodal data fusion in one embodiment of this application; Figure 2 This is a flowchart illustrating the determination of multiple key test scenarios in one embodiment of this application; Figure 3 This is a flowchart illustrating the acquisition of multimodal performance test data in one embodiment of this application; Figure 4 This is a schematic diagram of a psychological performance knowledge graph shown in one embodiment of this application. Detailed Implementation

[0020] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0021] Figure 1 This is a flowchart illustrating a method for psychological twin modeling of interview responses based on multimodal data fusion, as shown in one embodiment of this application. Figure 1 As shown, a method for modeling psychological twins of interview responses based on multimodal data fusion may include the following process.

[0022] S101, identify multiple key test scenarios and obtain multimodal performance test data of multiple test users in multiple key test scenarios.

[0023] Figure 2 This is a flowchart illustrating the determination of multiple key test scenarios in one embodiment of this application, such as... Figure 2 As shown, several key test scenarios have been identified, including: Determine a multimodal psychological performance factor set and multiple test scenarios, wherein the multimodal psychological performance factor set includes psychological performance factors corresponding to multiple modalities; Acquire multimodal performance test data from multiple test users across multiple test scenarios; For each test user and each test scenario, based on the test user's multimodal performance test data and multimodal psychological performance factor set in the test scenario, a multimodal response feature matrix corresponding to the test user's test scenario is constructed, where one row vector in the multimodal response feature matrix corresponds to one modality; For each test scenario, the psychological performance difference value corresponding to the test scenario is calculated based on the multimodal response feature matrix of each test user for the corresponding test scenario. Candidate test scenarios are determined based on the psychological performance differences corresponding to each test scenario. For any two candidate test scenarios, the psychological performance similarity between the two candidate test scenarios is calculated based on the multimodal response feature matrix of each test user corresponding to the two candidate test scenarios. Based on the similarity of psychological performance between any two candidate test scenarios, multiple key test scenarios are identified.

[0024] Specifically, the multimodal psychological performance factor set is a collection of psychological features extracted by integrating multi-dimensional data such as visual, auditory, linguistic, and behavioral data. It is used to quantify users' psychological response patterns in different testing scenarios. Visual modal psychological performance factors focus on facial expressions and eye movement characteristics, such as "micro-expression frequency" reflecting the degree of emotional inhibition, and "pupil dilation rate" associated with tension or excitement. Auditory modal factors cover the physiological and emotional attributes of sound, such as "voice fluctuation amplitude" revealing emotional stability, and "breathing frequency synchronicity" reflecting scene immersion. Linguistic modal factors emphasize semantics and expression methods, such as "proportion of negative words" reflecting emotional tendencies, and "response delay time" associated with cognitive load. Behavioral modal factors quantify behavioral characteristics through body movements and stress distribution, such as "seat pressure center trajectory deviation" reflecting postural anxiety, and "limb movement frequency" reflecting the degree of inhibition or activity. For example, in stress testing scenarios, users exhibiting high "micro-expression frequency" (frequent blinking, twitching of the mouth), low "tone fluctuation amplitude" (monotonous voice), high "negative vocabulary ratio" (frequent use of "difficult" or "unable"), and significant "seat pressure center trajectory shift" (leaning forward, frequent posture adjustments) may be classified as "high-stress sensitive." Their multimodal factor combination reveals psychological characteristics such as emotional inhibition, high cognitive load, and behavioral tension. This factor set overcomes the limitations of single-modality approaches through cross-modal data fusion.

[0025] Multiple testing scenarios can be designed to create a series of interview situations that induce different psychological states in test users. For example, a relaxed scenario can be set up where the interviewer starts with a friendly and easygoing attitude, asking some unpressured questions related to interesting life experiences, such as "Please share an interesting experience you had during your travels," to help test users relax. A serious thinking scenario would present more professional questions that require in-depth analysis, such as "What are your solutions to a certain technical problem that has recently emerged in the industry?" A tense scenario can be created by limiting the answering time and asking pointed questions ("What do you consider your biggest weakness, and how might it affect your performance in this position?").

[0026] Figure 3 This is a flowchart illustrating the acquisition of multimodal performance test data in one embodiment of this application, as shown below. Figure 3As shown, in some embodiments, acquiring multimodal performance test data of test users in key test scenarios includes: The visual modal response test data of test users in key test scenarios is obtained through an image acquisition device. The system uses a voice acquisition device to obtain auditory modal response test data and language modal response test data of test users in key test scenarios. By using a pressure acquisition device installed on the seat, behavioral modal response test data of the test users in key test scenarios are obtained.

[0027] Specifically, visual modal data is acquired through image acquisition devices such as cameras and depth sensors, primarily recording the nonverbal behaviors of test users in key test scenarios, including: Facial expressions: smiling, frowning, eye movement (such as avoidance or focus), dilated pupils (emotional fluctuations).

[0028] Body language: frequency of hand gestures (such as frequent hand rubbing when nervous), changes in sitting posture (leaning forward indicates engagement, leaning back indicates relaxation), and body turning (avoidance or openness).

[0029] Micro-expressions: brief (1 / 25 to 1 / 5 of a second) facial muscle movements that reflect true emotions (such as surprise or disgust); Auditory modal data is collected via microphone, and speech feature data includes acoustic parameters (such as pitch, volume, speech rate, pause frequency, etc.). Language modal response test data includes semantic information (such as keywords, logical structure, etc.) after text transcription.

[0030] Behavioral modal response test data is collected via seat pressure sensors, reflecting the test user's level of physical tension and movement patterns, including: Pressure distribution: Pressure values ​​in different areas of the seat (such as the back and buttocks) reflect the stability of the sitting posture; Stress changes: frequency (e.g., frequent changes in sitting posture) and amplitude (e.g., suddenly standing up); Action duration: The ratio of still time to action time (reflecting focus or anxiety).

[0031] In some embodiments, the pressure acquisition device includes a plurality of pressure sensing devices disposed at different locations on the seat; The placement of multiple pressure sensing devices is determined based on the following process: Identify multiple psychological triggering scenarios; For each psychologically induced scenario, seat pressure data of multiple test users in the psychologically induced scenario is obtained, where the seat pressure data includes pressure sequences at multiple positions of the seat; The improved butterfly optimization algorithm determines the placement of multiple pressure sensing devices based on seat pressure data from multiple test users in psychologically induced scenarios.

[0032] Specifically, psychologically induced scenarios refer to experimental conditions that actively trigger specific psychological states (such as stress, anxiety, relaxation, or deceptive intent) in test users through specific environmental design, task arrangement, or interaction methods. The core objective is to induce predictable psychological responses in subjects through controllable external stimuli. For example: Stress-induced scenarios: Anxiety or tension is triggered by time pressure, task difficulty, or social evaluation. Social anxiety triggering scenarios: Avoidance behaviors or stress states are triggered by simulating social evaluation or conflict situations; Relaxation-inducing scenarios: a calm or focused state can be triggered by a soothing environment or simplified tasks.

[0033] In the improved butterfly optimization algorithm, each butterfly represents a combination of positions sampled from multiple locations. For example, butterfly 1 represents positions 1-10, butterfly 2 represents positions 5-15, and so on.

[0034] In the improved butterfly optimization algorithm, the butterfly's fitness is determined based on the differentiated weights of the psychologically induced scenario, and the switching probability is determined based on the similarity of the current global optimal solution during iteration.

[0035] Specifically, the differentiated weights of psychological triggering scenarios can be determined through the following process: For each pressure sequence, features of the pressure sequence can be extracted, such as time-domain features (e.g., pressure mean, variance, maximum, minimum, etc.) and frequency-domain features (e.g., Fourier transform coefficients, power spectral density, etc.). For each psychologically induced scenario and each test user, a pressure feature matrix is ​​generated based on the test user's seat pressure data in the psychologically induced scenario. Each row vector of the pressure feature matrix corresponds to a position, and the row vectors of the pressure feature matrix are composed of the features of the pressure sequence of the position. For each psychologically induced scenario and any two test users, calculate the Euclidean distance between the two test users in the stress feature matrix of the psychologically induced scenario; For each psychological triggering scenario, the sum of the Euclidean distances between any two test users in the stress feature matrices of the psychological triggering scenario is calculated as the differential distance corresponding to the psychological triggering scenario. The ratio of the differential distance corresponding to the psychological triggering scenario to the sum of the differential distances corresponding to all psychological triggering scenarios is used as the differential weight of the psychological triggering scenario. For each butterfly and each psychologically induced scenario, based on the seat pressure data of the test user in the psychologically induced scenario, a pressure feature matrix of the corresponding butterfly for the test user in the psychologically induced scenario is generated. Here, one row vector of the pressure feature matrix corresponds to a position included by the butterfly, and the row vector of the pressure feature matrix is ​​composed of the features of the pressure sequence of the position included by the butterfly. For each butterfly, the mean of the stress feature matrix of the corresponding butterfly for each test user in the psychologically induced scenario is calculated as the stress feature matrix of the corresponding butterfly in the psychologically induced scenario. The Euclidean distance between the stress feature matrices of any two corresponding butterflies in the psychologically induced scenario is calculated. For each butterfly and each psychologically triggered scenario, calculate the Euclidean distance between the stress feature matrix of the psychologically triggered scenario and the corresponding butterfly of each other psychologically triggered scenario, and use it as the sum of the Euclidean distances of the corresponding butterfly of the psychologically triggered scenario; For each butterfly, the sum of the Euclidean distances corresponding to each psychological triggering scenario is weighted and summed based on the differentiated weights of each psychological triggering scenario, and this sum is used as the stress difference value for the corresponding butterfly.

[0036] The dependent variables of the fitness function of the improved butterfly optimization algorithm include the pressure difference value corresponding to the butterfly and the total number of positions included by the butterfly. The larger the pressure difference value corresponding to the butterfly and the smaller the total number of positions included by the butterfly, the greater the fitness of the butterfly.

[0037] For example, the fitness function of the improved butterfly optimization algorithm. It can be:

[0038] in, For the differentiated weight of the e-th psychological triggering scenario, Let Euclidean distance be the sum of the distances between the butterflies corresponding to the e-th psychological triggering scenario. The total number of psychologically induced scenarios. This represents the total number of locations included in the butterfly's range.

[0039] Understandably, firstly, for each psychologically induced scenario, features of the test users' seat pressure data are extracted, constructing a pressure feature matrix with position as the row vector. By calculating the sum of the Euclidean distances between the feature matrices of any two users in that scenario, the scenario's differential distance is obtained and normalized into differential weights, reflecting the degree of difference in stress performance among different users in that scenario—the larger the differential distance, the more significant the difference in stress response among users, requiring key positions to be selected; conversely, a smaller differential distance indicates that the scenario's impact on user stress is convergent, making position selection less necessary. Through differential weights, the algorithm prioritizes scenarios with significant differences in user stress responses, avoiding wasting computational resources on convergent scenarios, and ensuring that the selected positions accurately reflect individual stress characteristics, rather than generalizing ineffective positions.

[0040] Furthermore, for each butterfly, a stress feature matrix of its coverage location is generated, and the sum of Euclidean distances of the feature matrices for that butterfly under different scenarios is calculated. Then, the sum of Euclidean distances is weighted and summed based on the scenario's differentiated weights to obtain the stress difference value corresponding to the butterfly. The larger this value, the more effectively the butterfly's selected location combination can capture the differences in user stress changes under different scenarios. Simultaneously, the algorithm uses the total number of locations included in the butterfly as another fitness factor; the fewer the locations, the higher the fitness. Finally, the fitness function combines the stress difference value and the total number of locations, prioritizing butterfly combinations with strong difference capture capabilities and concise location selection. The weighted summation mechanism ensures that the butterfly combination location selection takes into account the differences in different scenarios, avoiding local optima in a single scenario, thereby improving the model's robustness under varying psychological induced conditions. Using the total number of locations as a fitness constraint minimizes the number of locations while ensuring difference capture capabilities, reducing the hardware cost and data complexity of subsequent stress monitoring.

[0041] The similarity to the current global optimum can be calculated iteratively based on the following process: For two consecutive iterations, calculate the proportion of overlap between the positions included in the current global optimal solution of the two iterations; Calculate the average of the overlap ratios of the positions included in the current global optimum in the most recent n (e.g., 10, 20, etc.) iterations, and use this as the similarity of the current global optimum.

[0042] The switching probability in the improved butterfly optimization algorithm can be determined based on the following formula:

[0043] in, The adjusted switching probability is dimensionless and ranges from 1 to 0. The basic switching probability is dimensionless and ranges from 1 to 0. This represents the similarity to the current global optimal solution.

[0044] Understandably, by calculating the proportion of overlapping positions of the global optimal solution in adjacent iterations, the stability of the search state is quantified. The average of the most recent n iterations is then used as the similarity score to reflect whether the algorithm has gotten stuck in a local optimum or continues to explore new regions. The above formula dynamically amplifies the base probability through similarity: when it is high (search stagnation), the switching probability is increased to strengthen global exploration; when it is low (search activity), the probability is decreased to focus on local development, thus balancing exploration and development, avoiding premature convergence and improving convergence efficiency. Its beneficial effect lies in achieving intelligent matching between switching probability and search state through the intuitive measure of position overlap, enhancing the algorithm's adaptability and robustness to complex optimization problems.

[0045] For each test user and each test scenario, based on the test user's multimodal performance test data and multimodal psychological performance factor set in the test scenario, a multimodal response feature matrix corresponding to the test user's test scenario is constructed. In this matrix, each row vector corresponds to a modality, and each element of the row vector is the feature value of a psychological performance factor corresponding to the modality.

[0046] For each test scenario, based on the multimodal response feature matrix of each test user corresponding to the test scenario, the psychological performance difference value corresponding to the test scenario is calculated. For example, the Euclidean distance between the multimodal response feature matrices of any two test users corresponding to the test scenario is calculated and summed. The sum of the Euclidean distances obtained is the psychological performance difference value corresponding to the test scenario.

[0047] Based on the psychological performance difference value corresponding to each test scenario, candidate test scenarios are determined. For example, the test scenarios are sorted from largest to smallest according to the psychological performance difference value, and the top k test scenarios (e.g., 5, 10, etc.) are selected as candidate test scenarios.

[0048] For any two candidate test scenarios, the psychological performance similarity between the two candidate test scenarios is calculated based on the multimodal response feature matrix of each test user corresponding to the two candidate test scenarios. For example, the mean of the multimodal response feature matrix of each test user corresponding to the candidate test scenario is calculated as the multimodal response feature matrix of the candidate test scenario. The cosine similarity of the multimodal response feature matrices of the two candidate test scenarios is then calculated as the psychological performance similarity between the two candidate test scenarios. Based on the psychological performance similarity between any two candidate test scenarios, multiple key test scenarios are determined. For example, if there are two candidate test scenarios whose psychological performance similarity is greater than the psychological performance similarity threshold (e.g., 0.6), then one of the candidate test scenarios is retained as the key test scenario.

[0049] This system integrates multi-dimensional behavioral and psychological data of test users across different scenarios. Based on matrix Euclidean distance calculation, it calculates the psychological performance differences between scenarios and selects the top k scenarios with the largest differences as candidates. This effectively focuses on test scenarios with significant user psychological changes and eliminates low-difference scenarios with redundant information. Furthermore, through matrix meanization and cosine similarity calculation, it quantifies the psychological performance similarity between candidate scenarios. Combined with threshold filtering, it retains scenarios with large differences and low similarity as key test scenarios, achieving hierarchical optimization of test scenarios: retaining core scenarios that fully reflect user psychological differences while avoiding scenario redundancy through similarity deduplication, thus improving testing efficiency and result representativeness.

[0050] S102, based on multimodal performance test data of multiple test users in multiple key test scenarios, determines multiple user types.

[0051] Specifically, it includes: For each test user and each key test scenario, a key multimodal response feature matrix for the test user corresponding to the key test scenario is constructed based on the test user's multimodal performance test data in the key test scenario. For any two test users, calculate the clustering distance between them based on the key multimodal response feature matrix corresponding to each key test scenario. By using clustering algorithms (such as K-means clustering, hierarchical clustering, etc.), multiple test users are clustered based on the clustering distance between any two test users to determine multiple user types.

[0052] Specifically, the test data of multimodal performance of test users in key test scenarios is transformed into a structured key multimodal response feature matrix. In this matrix, each row vector corresponds to a modality, and each row vector can include the feature value of each psychological performance factor corresponding to a modality. For example, for image data acquired from the visual modality, facial expression recognition algorithms are used to extract "micro-expression intensity index" (such as the weighted sum of the degree of eyebrow raising and the degree of mouth drooping) and "gaze focus dispersion" (the standard deviation of the time the gaze lingers in different areas of the screen) to quantify emotional arousal and attention allocation; after spectral analysis, speech signals acquired from the auditory modality are used to calculate "fundamental frequency fluctuation range" (the difference between the maximum and minimum values ​​of the speech fundamental frequency) and "breathing sound proportion" (the proportion of non-verbal vocalization time to the total speech time) to reflect the degree of physiological tension and emotional inhibition; language modality data are processed through natural language processing to extract "emotional load word density" (the frequency of occurrence of emotional words per 100 characters) and "semantic ambiguity index" (a sentence uncertainty score calculated based on word vectors) to reveal the intensity of subjective emotions and cognitive ambiguity; after dynamic tracking, seat pressure data from the behavioral modality is used to generate "pressure center offset speed" (the pixel distance of body center of gravity movement per second) and "local pressure peak frequency" (the number of times the hip / back pressure exceeds the threshold per minute) to characterize postural stability and muscle tension. Finally, the above features are classified and arranged according to modality to form a matrix structure in which rows represent users and columns contain four categories of features: visual, auditory, linguistic, and behavioral. For example, the matrix of a test user under stress may contain standardized values ​​such as "micro-expression intensity index = 0.72, fundamental frequency fluctuation range = 45Hz, emotional load word density = 18%, and stress center shift velocity = 2.3px / s". This matrix provides a structured psychological response pattern representation for subsequent cluster analysis by eliminating dimensional differences and redundancy between modalities.

[0053] For any two test users and each key test scenario, calculate the Euclidean distance between the key multimodal response feature matrices of the two test users for the corresponding key test scenarios. For any two test users, calculate the average of the Euclidean distances between the key multimodal response feature matrices of the two test users for the corresponding key test scenarios, and use this average as the cluster distance between the two test users.

[0054] Understandably, by integrating multimodal performance test data and extracting feature values ​​of psychological performance factors to form a structured feature matrix, the problem of fragmented information in a single modality is effectively solved, and the comprehensiveness of psychological state representation is improved. Secondly, by calculating the clustering distance between test users based on multimodal features, the differences in response patterns of different test users to the same scenario are quantified. Compared with traditional single-dimensional distance measurement, this can more accurately reflect the similarity of users' psychological characteristics. Finally, by using a clustering algorithm, multiple test users are clustered according to the clustering distance between any two test users to determine multiple user types. The construction of cross-modal feature matrices enhances the richness of psychological state description and avoids classification bias caused by modality loss. Through multimodal data fusion and unsupervised learning, the objectivity and accuracy of user classification are significantly improved.

[0055] S103, Obtain multimodal interview response data from multiple test users.

[0056] The multimodal interview response data of the test users includes visual modal interview response data, auditory modal interview response data, verbal modal interview response data, and behavioral modal interview response data. The method for obtaining the multimodal interview response data of the test users is similar to the method for obtaining the multimodal performance test data of the test users in key test scenarios, and will not be repeated here.

[0057] S104 constructs a psychological performance knowledge graph based on multimodal interview response data from multiple test users.

[0058] in, Figure 4 This is a schematic diagram of a psychological performance knowledge graph shown in one embodiment of this application, such as... Figure 4 As shown, the psychological performance knowledge graph is used to record multiple key psychological performance factors for different user types.

[0059] In some embodiments, S104 specifically includes: For each user type, based on the key multimodal response feature matrix of the test users included in the user type for each key test scenario, calculate the factor performance difference value of each psychological performance factor corresponding to the user type, and determine multiple key psychological performance factors of the user type based on the factor performance difference value of each psychological performance factor corresponding to the user type. A knowledge graph of psychological performance is constructed based on multiple key factors of psychological performance for each user type.

[0060] In some embodiments, based on the key multimodal response feature matrix corresponding to each key test scenario for the test users included in the user types, the factor performance difference value for each psychological performance factor corresponding to the user type is calculated, including: For any two key test scenarios, based on the key multimodal response feature matrix of the test users included in the user type corresponding to the two key test scenarios, calculate the factor performance difference value of the psychological performance factor corresponding to the two key test scenarios. For example, the absolute value of the difference of the feature values ​​of the psychological performance factor of each test user included in the user type can be calculated under the two key test scenarios, and the sum of the absolute values ​​of the difference of the feature values ​​of the psychological performance factor of each test user included in the user type can be used as the factor performance difference value of the psychological performance factor corresponding to the two key test scenarios. Based on the factor performance difference value of any two key test scenarios corresponding to the psychological performance factor, the factor performance difference value of the user type corresponding to the psychological performance factor can be calculated. For example, the factor performance difference values ​​of any two key test scenarios corresponding to the psychological performance factor can be summed as the factor performance difference value of the user type corresponding to the psychological performance factor.

[0061] Specifically, psychological performance factors can be ranked from largest to smallest based on the difference in factor performance values, and the top n psychological performance factors (e.g., 3, 5, etc.) can be selected as key psychological performance factors for user types.

[0062] Understandably, multi-scenario comparative analysis effectively eliminates factor misjudgments caused by the randomness of a single scenario, ensuring that key factors can stably reflect the psychological characteristics of user types. Secondly, factor screening based on user type makes the knowledge graph closely aligned with the behavioral patterns of different user groups, improving the accuracy and personalization of psychological assessments. Furthermore, data-driven difference value quantification avoids interference from subjective experience in factor selection, enhancing the objectivity and interpretability of key factor identification. Finally, the constructed knowledge graph integrates the relationship between user types and key factors in a structured form.

[0063] S105: Obtain multimodal performance test data of the interviewee in multiple key test scenarios to determine the user type to which the interviewee belongs.

[0064] Specifically, it includes: Based on the multimodal performance test data of interviewees in multiple key test scenarios, a key multimodal response feature matrix for each key test scenario is constructed. For each user type, based on the key multimodal response feature matrix of the test users included in the user type for each key test scenario, construct the key multimodal response feature matrix for each user type for each key test scenario; Based on the key multimodal response feature matrix of the interviewee for each key test scenario and the key multimodal response feature matrix of each user type for each key test scenario, the user type to which the interviewee belongs is determined.

[0065] Specifically, for each user type and each key test scenario, the mean of the key multimodal response feature matrix of the test users corresponding to the key test scenario for each user type is calculated to obtain the key multimodal response feature matrix of the key test scenario for each user type.

[0066] For each user type and each key test scenario, calculate the Euclidean distance between the key multimodal response feature matrix of the interviewee corresponding to the key test scenario and the key multimodal response feature matrix of the user type corresponding to the key test scenario.

[0067] For each user type, the mean of the Euclidean distance between the key multimodal response feature matrix of the interviewee for each key test scenario and the key multimodal response feature matrix of the user type for each key test scenario is calculated to obtain the interviewee's attribution value to the user type. The smaller the mean of the Euclidean distance, the greater the interviewee's attribution value to the user type.

[0068] The user type to which the interviewee belongs is the user type with the highest user type affiliation value.

[0069] Understandably, firstly, constructing key multimodal response feature matrices for each key test scenario for interviewees and user types comprehensively and meticulously depicts the interviewees' performance characteristics in different scenarios, providing rich evidence for accurate classification. Calculating the mean of the test user matrix for each user type to construct its scenario matrix effectively integrates group characteristics, eliminates interference from individual differences, and improves the stability and reliability of classification. Secondly, calculating the Euclidean distance between the interviewee and the scenario matrices of each user type, and further calculating the mean to obtain the attribution value, makes the classification more scientific and objective. Euclidean distance intuitively reflects the similarity between the interviewee and the user type feature matrix; the smaller the mean distance, the closer the interviewee is to the characteristics of that user type, and the larger the attribution value, thus accurately determining the interviewee's type. Finally, the user type with the largest attribution value is determined as the interviewee's type, ensuring the accuracy and relevance of the classification results.

[0070] S106, Obtain the interviewee's multimodal interview response data.

[0071] Specifically, the methods for obtaining multimodal interview response data from interviewees are similar to those for obtaining multimodal interview response data from test users, and will not be elaborated here.

[0072] S107, based on the interviewee's multimodal interview response data, the interviewee's user type, and the psychological performance knowledge graph, conducts psychological twin modeling of the interviewee's responses.

[0073] Specifically, it includes: For each user type, a response mental twin model is constructed based on the psychological performance knowledge graph. The input of the response mental twin model is the feature value of the key psychological performance factor of the user type recorded in the psychological performance knowledge graph. The output of the response mental twin model is the interviewee's response psychology. The response mental twin model can be a long short-term memory network model. Retrieve the corresponding psychological twin model of the interviewee's user type, and perform psychological twin modeling of the interviewee's responses based on the interviewee's multimodal interview response data.

[0074] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of the embodiments described herein are intended to be illustrative rather than limiting, and should be considered consistent with the teachings of this specification. Accordingly, the embodiments described herein are not limited to those explicitly introduced and described herein.

Claims

1. A method for psychological twin modeling of interview responses based on multimodal data fusion, characterized in that, include: Identify multiple key test scenarios and obtain multimodal performance test data from multiple test users in these scenarios. Based on multimodal performance test data from multiple test users in multiple key test scenarios, multiple user types were identified. Obtain multimodal interview response data from multiple test users; Based on multimodal interview response data from multiple test users, a psychological performance knowledge graph is constructed, which is used to record multiple key psychological performance factors for different user types. Obtain multimodal performance test data of interviewees in multiple key test scenarios to determine the user type to which the interviewees belong; Obtain multimodal interview response data from interviewees; Based on the interviewee's multimodal interview response data, the interviewee's user type, and the psychological performance knowledge graph, a psychological twin model of the interviewee's response is created.

2. The method for psychological twin modeling of interview responses based on multimodal data fusion according to claim 1, characterized in that, Several key test scenarios were identified, including: Determine a multimodal psychological performance factor set and multiple test scenarios, wherein the multimodal psychological performance factor set includes psychological performance factors corresponding to multiple modalities; Acquire multimodal performance test data from multiple test users across multiple test scenarios; For each test user and each test scenario, based on the test user's multimodal performance test data and multimodal psychological performance factor set in the test scenario, a multimodal response feature matrix corresponding to the test user's test scenario is constructed, where one row vector in the multimodal response feature matrix corresponds to one modality; For each test scenario, the psychological performance difference value corresponding to the test scenario is calculated based on the multimodal response feature matrix of each test user for the corresponding test scenario. Candidate test scenarios are determined based on the psychological performance differences corresponding to each test scenario. For any two candidate test scenarios, the psychological performance similarity between the two candidate test scenarios is calculated based on the multimodal response feature matrix of each test user corresponding to the two candidate test scenarios. Based on the similarity of psychological performance between any two candidate test scenarios, multiple key test scenarios are identified.

3. The method for psychological twin modeling of interview responses based on multimodal data fusion according to claim 2, characterized in that, Obtain multimodal performance test data of test users in key test scenarios, including: The visual modal response test data of test users in key test scenarios is obtained through an image acquisition device. The system uses a voice acquisition device to obtain auditory modal response test data and language modal response test data of test users in key test scenarios. By using a pressure acquisition device installed on the seat, behavioral modal response test data of the test users in key test scenarios are obtained.

4. The method for psychological twin modeling of interview responses based on multimodal data fusion according to claim 3, characterized in that, The pressure acquisition device includes multiple pressure sensors installed at different locations on the seat. The placement of multiple pressure sensing devices is determined based on the following process: Identify multiple psychological triggering scenarios; For each psychologically induced scenario, seat pressure data of multiple test users in the psychologically induced scenario is obtained, where the seat pressure data includes pressure sequences at multiple positions of the seat; The improved butterfly optimization algorithm determines the placement of multiple pressure sensing devices based on seat pressure data from multiple test users in psychologically induced scenarios.

5. The method for psychological twin modeling of interview responses based on multimodal data fusion according to claim 4, characterized in that, In the improved butterfly optimization algorithm, the butterfly's fitness is determined based on the differentiated weights of the psychologically induced scenario, and the switching probability is determined based on the similarity to the current global optimal solution during iteration.

6. A method for psychological twin modeling of interview responses based on multimodal data fusion according to any one of claims 3-5, characterized in that, Based on multimodal performance test data from multiple test users across multiple key test scenarios, several user types were identified, including: For each test user and each key test scenario, a key multimodal response feature matrix for the test user corresponding to the key test scenario is constructed based on the test user's multimodal performance test data in the key test scenario. For any two test users, calculate the clustering distance between them based on the key multimodal response feature matrix corresponding to each key test scenario. By using a clustering algorithm, multiple test users are clustered based on the clustering distance between any two test users, thus determining multiple user types.

7. The method for psychological twin modeling of interview responses based on multimodal data fusion according to claim 6, characterized in that, The multimodal interview response data of the test users includes visual modal interview response data, auditory modal interview response data, verbal modal interview response data, and behavioral modal interview response data; Based on multimodal interview response data from multiple test users, a psychological performance knowledge graph was constructed, including: For each user type, based on the key multimodal response feature matrix of the test users included in the user type for each key test scenario, calculate the factor performance difference value of each psychological performance factor corresponding to the user type, and determine multiple key psychological performance factors of the user type based on the factor performance difference value of each psychological performance factor corresponding to the user type. A psychological performance knowledge graph is constructed based on multiple key psychological performance factors for each user type.

8. The method for psychological twin modeling of interview responses based on multimodal data fusion according to claim 7, characterized in that, Based on the key multimodal response feature matrix of test users corresponding to each key test scenario, including user types, the factor performance difference value of each psychological performance factor corresponding to user type is calculated, including: For any two key test scenarios, based on the key multimodal response feature matrix of the test users corresponding to the two key test scenarios, including the user types, the factor performance difference value of the psychological performance factor corresponding to the two key test scenarios is calculated. Based on the difference in factor performance between any two key test scenarios corresponding to the psychological performance factor, the difference in factor performance between user types corresponding to the psychological performance factor is calculated.

9. A method for psychological twin modeling of interview responses based on multimodal data fusion according to claim 5, characterized in that, Determine the user type to which the interviewee belongs, including: Based on the multimodal performance test data of interviewees in multiple key test scenarios, a key multimodal response feature matrix for each key test scenario is constructed. For each user type, based on the key multimodal response feature matrix of the test users included in the user type for each key test scenario, construct the key multimodal response feature matrix for each user type for each key test scenario; Based on the key multimodal response feature matrix of the interviewee for each key test scenario and the key multimodal response feature matrix of each user type for each key test scenario, the user type to which the interviewee belongs is determined.

10. A method for psychological twin modeling of interview responses based on multimodal data fusion according to any one of claims 1-5, characterized in that, Based on the interviewees' multimodal interview response data, the interviewees' user types, and psychological performance knowledge graphs, a psychological twin model of the interviewees' responses is constructed, including: For each user type, a psychological twin model of responses corresponding to the user type is constructed based on the psychological performance knowledge graph; Retrieve the corresponding psychological twin model of the interviewee's user type, and perform psychological twin modeling of the interviewee's responses based on the interviewee's multimodal interview response data.

Images