User cooperation and interaction capability assessment method, system and terminal based on multi-modal data

Abstract

The invention discloses a multi-modal data-based user cooperation and interaction capability evaluation method, system and terminal, and the method comprises the steps: designing a team task under a pressure situation, selecting a plurality of users to participate in the team task, and obtaining the multi-modal data of the plurality of users in the team task; performing feature extraction on the multi-modal data to obtain a multi-modal intermediate variable, and inputting the multi-modal intermediate variable into a trained multi-modal intermediate variable analysis model for feature extraction to obtain a current multi-modal feature; and inputting the current multi-modal features into a trained multi-modal fusion and evaluation model to carry out multi-feature fusion and evaluation to obtain respective cooperation and interaction capability evaluation results of the plurality of users. According to the method, an automatic evaluation framework with two-stage verification is constructed, and the accuracy and reliability of user cooperation and interaction ability evaluation are greatly improved through a training and verification mechanism with double introduction of expert evaluation conclusions and psychological measurement data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a user collaboration and interaction ability evaluation method and system based on multi-modal data and a terminal. BACKGROUND

[0002] Group dynamics is a scientific field dedicated to studying group behavior and processes, aiming to explore the nature, development laws of groups, and their relationships with individuals, other groups, and larger institutions. It focuses on various dynamic processes in groups, such as interactions between members, formation and changes of group structure, and the influence of groups on members.

[0003] User collaboration and interaction ability is a multi-dimensional comprehensive ability system. The core is the ability of individuals to effectively communicate, coordinate, and empathize with others in the process of jointly advancing goals, solving problems, or completing tasks, achieving efficient linkage and value co-creation.

[0004] However, most existing evaluation methods fail to fully utilize multi-modal data in team tasks, such as voice tone, facial expressions, gestures, and other non-verbal information. These data are important clues to understanding interpersonal interactions and emotional states. Existing technologies fail to extract and use multi-modal data for human-computer interaction and team collaboration and interaction ability evaluation, which greatly reduces the depth and accuracy of evaluation.

[0005] Therefore, the existing technology still needs to be improved and developed. SUMMARY

[0006] The main purpose of the present application is to provide a user collaboration and interaction ability evaluation method, system, terminal and computer readable storage medium based on multi-modal data, aiming to solve the problem that the existing technology fails to fully utilize multi-modal data in team tasks to understand interpersonal interactions and emotional states, resulting in inaccurate evaluation results for user collaboration and interaction ability.

[0007] To achieve the above purpose, the present application provides a user collaboration and interaction ability evaluation method based on multi-modal data, which comprises the following steps: Design a team task under pressure situation, select multiple users to participate in the team task, and obtain multi-modal data of the multiple users in the team task, including facial expression data, voice data, and posture data; Feature extraction is performed on the multimodal data to obtain multimodal intermediate variables. The multimodal intermediate variables are then input into a trained multimodal intermediate variable analysis model for feature extraction to obtain current multimodal features. The current multimodal features include current emotional state, current emotional intensity, current leadership ability, current collaboration style, current posture focus, and current task participation. The current multimodal features are input into the trained multimodal fusion and evaluation model to perform multimodal feature fusion and evaluation, thereby obtaining the evaluation results of the collaboration and interaction capabilities of each of the users.

[0008] Optionally, the method for evaluating user collaboration and interaction capabilities based on multimodal data, wherein designing a team task under stress, selecting multiple users to participate in the team task, and acquiring multimodal data of multiple users in the team task, specifically includes: Based on gamification theory and stressor theory, the team task was designed, and multiple users were selected to participate in the team task. Record all video and audio data in the team's tasks in real time, and collect multimodal data of each user from all the video and audio data.

[0009] Optionally, in the method for evaluating user collaboration and interaction capabilities based on multimodal data, the multimodal intermediate variables include: facial key points, confidence scores for specific emotions, and number of emotion category changes corresponding to the facial expression data; speech rate, pitch, fluctuation amplitude, number of positive words, number of negative words, speaking order, interruption frequency, and speaking duration distribution corresponding to the speech data; and body orientation, gesture frequency, and posture type corresponding to the posture data.

[0010] Optionally, the method for evaluating user collaboration and interaction capabilities based on multimodal data, wherein the step of extracting features from the multimodal data to obtain multimodal intermediate variables specifically includes: The facial expression data was feature extracted using the MediaPipe facial model to obtain facial key points and confidence scores for specific emotions, and the number of changes in emotion categories was analyzed in conjunction with a specific time window. The speech data is digitally processed to quantify the user's speech rate, pitch, and fluctuation range. The speech data is then transcribed to obtain speech text. Natural language processing technology is used to perform sentiment analysis on the speech text to obtain the number of positive words and the number of negative words. The speech data is identified using the TalkNet model, and the speaking duration and speaking time distribution are quantified. The user's speaking order, interruption frequency, and speaking duration distribution are then extracted from the speaking duration and speaking time distribution. The posture data is analyzed frame by frame using the YOLO Pose model to obtain the position coordinates of multiple posture key points. Based on the position coordinates, the user's body orientation, gesture frequency, and posture type are obtained.

[0011] Optionally, the method for evaluating user collaboration and interaction capabilities based on multimodal data, wherein the step of inputting the multimodal intermediate variables into a trained multimodal intermediate variable analysis model for feature extraction to obtain the current multimodal features, further includes: Collect historical multimodal intermediate variables from historical team tasks, and obtain the multimodal features labeled by experts for the historical multimodal intermediate variables; A multimodal intermediate variable analysis model is constructed, using the historical multimodal intermediate variables as data samples and the multimodal features as labels; The multimodal intermediate variable analysis model is trained using the data samples and the labels to obtain a trained multimodal intermediate variable analysis model.

[0012] Optionally, in the method for evaluating user collaboration and interaction capabilities based on multimodal data, the multimodal features include emotional state, emotional intensity, leadership ability, collaboration style, postural focus, and task participation. The emotional state and the emotional intensity are obtained by labeling intermediate variables corresponding to historical facial expression data; The leadership ability and the collaboration style are obtained by labeling intermediate variables corresponding to historical voice data; The posture focus and the task engagement are obtained by labeling intermediate variables corresponding to historical posture data.

[0013] Optionally, the method for evaluating user collaboration and interaction capabilities based on multimodal data, wherein the step of inputting the current multimodal features into a trained multimodal fusion and evaluation model for multi-feature fusion and evaluation to obtain the collaboration and interaction capability evaluation results for each of the users, further includes: Historical multimodal intermediate variables are input into a trained multimodal intermediate variable analysis model for feature extraction to obtain historical multimodal features; A multimodal fusion and evaluation model based on an attention mechanism is constructed, the multimodal fusion and evaluation model including a multimodal feature fusion module and a multimodal feature evaluation module; The historical multimodal features are input into the multimodal fusion and evaluation model for vectorization to obtain vectorized features. The multimodal feature fusion module assigns weights to the vectorized features under different task area scenarios and performs attention-weighted fusion based on the assigned weights to obtain a fused feature vector. The user's collaboration and interaction abilities are assessed using a psychometric scale, and the assessed results are used as labels and the fused feature vector is used as a data sample. The data samples are input into the multimodal feature evaluation module for evaluation and prediction, and the predicted evaluation results of collaboration and interaction capabilities are output. Based on the predicted evaluation results and the labels, the multimodal fusion and evaluation model is trained and optimized to obtain a trained multimodal fusion and evaluation model.

[0014] Furthermore, to achieve the above objectives, the present invention also provides a user collaboration and interaction capability assessment system based on multimodal data, wherein the user collaboration and interaction capability assessment system based on multimodal data includes: A multimodal data acquisition module is used to design team tasks under stress, select multiple users to participate in the team tasks, and acquire multimodal data of the multiple users in the team tasks. The multimodal data includes facial expression data, voice data, and posture data. The multimodal feature extraction module is used to extract features from the multimodal data to obtain multimodal intermediate variables. The multimodal intermediate variables are then input into a trained multimodal intermediate variable analysis model for feature extraction to obtain the current multimodal features. The current multimodal features include the current emotional state, current emotional intensity, current leadership ability, current collaboration style, current posture focus, and current task participation. The multimodal feature fusion and evaluation module is used to input the current multimodal features into the trained multimodal fusion and evaluation model to perform multi-feature fusion and evaluation, and obtain the evaluation results of the collaboration and interaction capabilities of each of the users.

[0015] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a user collaboration and interaction capability assessment program based on multimodal data stored in the memory and executable on the processor, wherein when the user collaboration and interaction capability assessment program based on multimodal data is executed by the processor, it implements the steps of the user collaboration and interaction capability assessment method based on multimodal data as described above.

[0016] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a user collaboration and interaction capability assessment program based on multimodal data, and when the user collaboration and interaction capability assessment program based on multimodal data is executed by a processor, it implements the steps of the user collaboration and interaction capability assessment method based on multimodal data as described above.

[0017] In this invention, a team task under stress is designed, multiple users are selected to participate in the task, and multimodal data of the users in the task is acquired. This multimodal data includes facial expression data, voice data, and posture data. Feature extraction is performed on the multimodal data to obtain multimodal intermediate variables. These intermediate variables are then input into a trained multimodal intermediate variable analysis model for feature extraction, yielding current multimodal features. These current multimodal features include current emotional state, current emotional intensity, current leadership ability, current collaboration style, current posture focus, and current task participation. These current multimodal features are then input into a trained multimodal fusion and evaluation model for multi-feature fusion and evaluation, resulting in evaluation results for each user's individual collaboration and interaction abilities. This invention constructs a two-stage validated automated evaluation framework. Through a training and validation mechanism that incorporates both expert evaluation conclusions and psychometric data, the accuracy and reliability of user collaboration and interaction ability evaluation are greatly improved. Attached Figure Description

[0018] Figure 1 This is a flowchart of a preferred embodiment of the user collaboration and interaction capability assessment method based on multimodal data of the present invention; Figure 2 This is a schematic diagram illustrating the principle of the user collaboration and interaction capability assessment method based on multimodal data of the present invention; Figure 3 This is a structural diagram of a preferred embodiment of the user collaboration and interaction ability assessment system based on multimodal data of the present invention; Figure 4 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0019] This application provides a method, system, and terminal for evaluating user collaboration and interaction capabilities based on multimodal data. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0020] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0021] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0022] The preferred embodiment of the user collaboration and interaction capability assessment method based on multimodal data described in this invention, such as... Figure 1 and Figure 2 As shown, the method for evaluating user collaboration and interaction capabilities based on multimodal data includes the following steps: Step S10: Design a team task under stress, select multiple users to participate in the team task, and obtain multimodal data of the multiple users in the team task.

[0023] Specifically, based on gamification theory and stressor theory, the team task was designed, and multiple users were selected to participate in the team task.

[0024] In this embodiment, based on the theoretical framework of the three constructs of group dynamics, combined with relevant theoretical research on gamification and stressor theory, this invention first designs a team task under stress. This task is characterized by its structured nature, high simulation, high immersion, and strong interactivity, effectively assessing and training teamwork and interaction abilities, and stimulating participants' authentic interpersonal interaction styles. This invention specifically incorporates different types of stressors, such as time pressure, task load, and role conflict, into the task design. By carefully observing participants' communication, division of labor, and problem-solving behaviors under these stressful environments, a scientific assessment of their interpersonal interaction and collaboration abilities is achieved.

[0025] This invention incorporates gamification theory into team task design, enhancing participants' immersion and motivation by integrating gamification elements into team tasks. This aims to more effectively stimulate interpersonal interaction and collaborative behavior. Leveraging research findings on how gamification improves positive outcomes, the invention uses gamification design principles such as clear goals and immediate feedback to evoke intrinsic motivation and create a flow experience, allowing members to become fully immersed in the task. By utilizing gamification elements such as social interaction, narrative, time, and stages, the invention promotes deep collaboration among members. In achieving the task, participation, collaborative behavior, and task performance are enhanced. This makes team tasks not only fun and immersive but also valuable for assessing interpersonal interaction types, collaboration, and interaction abilities, as well as developing communication, collaboration, and interaction skills.

[0026] Furthermore, all video and audio data in the team task are recorded in real time, and multimodal data of multiple users are collected from all the video and audio data.

[0027] It is understood that this embodiment is equipped with a dedicated data acquisition system. This system integrates multimodal information processing technologies such as scale assessment, facial expression recognition, and voice analysis to collect multimodal data, including facial expression data, voice data, and posture data. This provides a more accurate and objective selection criterion for interpersonal interaction and teamwork.

[0028] To achieve simultaneous acquisition of multiple video and audio files, this embodiment developed a related acquisition program. This program employs a multi-threaded architecture, handling video capture, audio capture, video saving, and audio saving separately. Each camera and audio device has an independent thread responsible for data acquisition, while the saving thread is responsible for synchronously writing the acquired data to a file. The system will record all video and audio from team members during task execution and discussions, thus obtaining multimodal data from multiple users.

[0029] Step S20: Extract features from the multimodal data to obtain multimodal intermediate variables. Input the multimodal intermediate variables into the trained multimodal intermediate variable analysis model for feature extraction to obtain the current multimodal features. The current multimodal features include the current emotional state, current emotional intensity, current leadership ability, current collaboration style, current posture focus, and current task participation.

[0030] In this embodiment, the multimodal intermediate variables include: facial key points, confidence scores for specific emotions, and number of emotion category changes corresponding to the facial expression data; speech rate, pitch, fluctuation amplitude, number of positive words, number of negative words, speaking order, interruption frequency, and speaking duration distribution corresponding to the speech data; and body orientation, gesture frequency, and posture type corresponding to the posture data.

[0031] The step of extracting features from the multimodal data to obtain multimodal intermediate variables specifically includes: The facial expression data was feature extracted using the MediaPipe facial model to obtain facial key points and confidence scores for specific emotions, and the number of changes in emotion categories was analyzed in conjunction with a specific time window. The speech data is digitally processed to quantify the user's speech rate, pitch, and fluctuation range. The speech data is then transcribed to obtain speech text. Natural language processing technology is used to perform sentiment analysis on the speech text to obtain the number of positive words and the number of negative words. The speech data is identified using the TalkNet model, and the speaking duration and speaking time distribution are quantified. The user's speaking order, interruption frequency, and speaking duration distribution are then extracted from the speaking duration and speaking time distribution.

[0032] Understandably, this application utilizes deep learning models or mathematical methods from their respective fields to model and analyze the video and audio data collected during the task, in order to extract a series of intermediate features. Regarding facial expression data analysis, the MediaPipe facial model (a set of machine learning models for facial detection and analysis, characterized by high efficiency, accuracy, and lightweight nature, suitable for various platforms and application scenarios) is deployed to extract probability values ​​or confidence scores of facial key points and specific emotions, used to quantify the intensity of emotional expression, and combined with a specific time window to analyze the number of changes in emotion category.

[0033] In terms of speech data analysis, we used the TalkNet model (a deep learning-based model with important applications in speech synthesis and active speaker detection) to identify speakers in audio and video, quantifying the speaking duration and distribution of each person to extract features such as the speaking order, interruption frequency, and speaking duration distribution among team members. Simultaneously, we used mathematical methods to digitize the audio features to quantify each member's speech rate, pitch, and fluctuation range. Furthermore, we applied natural language processing techniques to perform sentiment analysis on the transcribed text, calculating the frequency of positive and negative words.

[0034] In terms of posture data analysis, the posture and hand posture of the members are analyzed by using the YOLO Pose model (the YOLO Pose model is a series of posture estimation models based on the YOLO framework, which can quickly and accurately detect human key points in images or videos, thereby realizing the recognition and analysis of human posture) to obtain the position coordinates of the posture key points frame by frame, and then use these position coordinates to calculate indicators such as body orientation, gesture frequency and type.

[0035] Furthermore, the facial key points, specific emotion confidence scores, and emotion category change frequency corresponding to the facial expression data, the speech rate, pitch, fluctuation amplitude, number of positive words, number of negative words, speech order, interruption frequency, and speech duration distribution corresponding to the speech data, and the body orientation, gesture frequency, and posture type corresponding to the posture data are input into a trained multimodal intermediate variable analysis model for feature extraction to obtain the current multimodal features.

[0036] Furthermore, the multimodal intermediate variables are input into a trained multimodal intermediate variable analysis model for feature extraction to obtain the current multimodal features. This process also includes: Collect historical multimodal intermediate variables from historical team tasks, and obtain the multimodal features labeled by experts for the historical multimodal intermediate variables; A multimodal intermediate variable analysis model is constructed, using the historical multimodal intermediate variables as data samples and the multimodal features as labels; The multimodal intermediate variable analysis model is trained using the data samples and the labels to obtain a trained multimodal intermediate variable analysis model.

[0037] The multimodal features include emotional state, emotional intensity, leadership ability, collaboration style, postural focus, and task engagement. The emotional state and emotional intensity are obtained by labeling intermediate variables corresponding to historical facial expression data. The leadership ability and collaboration style are obtained by labeling intermediate variables corresponding to historical speech data. The postural focus and task engagement are obtained by labeling intermediate variables corresponding to historical posture data.

[0038] Specifically, this application establishes a multimodal intermediate variable analysis model, trained using expert-annotated data. Experts, while watching task videos, annotate the emotional state and intensity of members at specific moments (e.g., "When member A completes the task, member B's 'joy' expression intensity is 'high'"). The multimodal intermediate variable analysis model then outputs the emotional state and intensity of each member in a specific scenario based on facial key points extracted in the first stage and confidence scores for each expression, using expert-annotated data as ground truth for training. Similarly, experts annotate the communication intent and style of key dialogue segments (e.g., "efficient collaboration," "leader-led," or "conflict arising"), and assign leadership and collaboration style indicators to each member based on voice (e.g., frequently interrupting others, speaking in a high tone, and speaking quickly indicates strong leadership; positive language indicates a greater willingness to collaborate). Features extracted from voice analysis are input into the multimodal intermediate variable analysis model for training and validation. Regarding body posture, experts assess indicators such as each member's postural focus and engagement to evaluate the effectiveness of nonverbal communication and task commitment.

[0039] As can be seen, this invention first utilizes a deep learning model to extract a series of key intermediate variables from the raw data. To ensure the validity and reliability of the model, expert evaluation conclusions are embedded into these intermediate variables. These expert conclusions will be used as labeled data to train and validate the model in the first part, ensuring that the intermediate variables extracted by the model are highly consistent with the cognition of human experts.

[0040] Step S30: Input the current multimodal features into the trained multimodal fusion and evaluation model to perform multimodal fusion and evaluation, and obtain the evaluation results of the collaboration and interaction capabilities of each of the users.

[0041] Specifically, the multimodal fusion and evaluation model includes a multimodal feature fusion module and a multimodal feature evaluation module. The current multimodal features are input into the multimodal feature fusion module of the trained multimodal fusion and evaluation model for attention-weighted fusion, resulting in a current fused feature vector. This current fused feature vector is then input into the multimodal feature evaluation module of the trained multimodal fusion and evaluation model for evaluation, yielding individual evaluation results for the collaboration and interaction capabilities of multiple users. Based on these evaluation results, a detailed evaluation report is generated and visualized for easy user understanding.

[0042] Furthermore, the step of inputting the current multimodal features into a trained multimodal fusion and evaluation model to perform multi-feature fusion and evaluation, and obtaining evaluation results of the collaboration and interaction capabilities of each of the users, further includes: Historical multimodal intermediate variables are input into a trained multimodal intermediate variable analysis model for feature extraction to obtain historical multimodal features; A multimodal fusion and evaluation model based on an attention mechanism is constructed, the multimodal fusion and evaluation model including a multimodal feature fusion module and a multimodal feature evaluation module; The historical multimodal features are input into the multimodal fusion and evaluation model for vectorization to obtain vectorized features. The multimodal feature fusion module assigns weights to the vectorized features under different task area scenarios and performs attention-weighted fusion based on the assigned weights to obtain a fused feature vector. The user's collaboration and interaction abilities are assessed using a psychometric scale, and the assessed results are used as labels and the fused feature vector is used as a data sample. The data samples are input into the multimodal feature evaluation module for evaluation and prediction, and the predicted evaluation results of collaboration and interaction capabilities are output. Based on the predicted evaluation results and the labels, the multimodal fusion and evaluation model is trained and optimized to obtain a trained multimodal fusion and evaluation model.

[0043] In this embodiment, the final quantification of team collaboration ability is achieved through advanced multimodal data fusion and evaluation, relying on advanced machine learning models and combining psychometric data for training and validation.

[0044] In the implementation of the multimodal data fusion model, this invention adopts a deep learning model based on an attention mechanism as the core fusion architecture. This model can intelligently learn and assign weights to data from different modalities, overcoming the limitations of simple splicing of modal information in traditional methods. Specifically, it first vectorizes all intermediate variables extracted in the first stage; then, it designs a cross-modal attention layer within the model, allowing the model to automatically determine which modal information is most critical in a specific task context. For example, it assigns higher weights to posture and facial expression vectors in the "silent task," while increasing the weight of speech vectors in the "second task." Finally, after attention weighting, the vectors from different modalities are spliced ​​or added to form a unified fusion feature vector, and a fully connected layer is used for feature fusion and dimensionality reduction.

[0045] In the implementation and validation of the evaluation result generator, supervised learning is employed, using real psychometric data to train and validate the model. Specifically, the fused feature vector is used as the model input, while the interpersonal interaction and collaboration ability data of each participant obtained from the psychometric scales are used as the model output labels. A regression model (such as support vector regression) is used to predict continuous collaboration ability scores, and a classification model is used to evaluate the personality type of each member. To ensure the scientific rigor and validity of the evaluation, the trained model is validated using an independent psychometric dataset. The training of the multimodal fusion and evaluation model is completed by calculating the correlation or accuracy between the model's predicted values ​​and the real scale data.

[0046] As can be seen, this invention constructs a multimodal fusion and assessment model. This model aims to integrate and transform previously extracted intermediate variables into quantifiable indicators of interpersonal interaction and collaboration abilities. Utilizing the multimodal fusion model, data from different modalities are intelligently weighted and fused, and a causal inference model is employed to identify deep causal relationships between variables. Furthermore, the multimodal fusion and assessment model is trained and validated based on interpersonal interaction and collaboration ability data obtained from psychometric scales for each participant. This ensures the scientific validity and effectiveness of the final assessment results.

[0047] In summary, to improve the evaluation effectiveness of interpersonal interaction and collaboration abilities and achieve automated evaluation, this invention proposes a user collaboration and interaction ability evaluation method integrating facial expression recognition, speech analysis, and posture analysis for designed team tasks, constructing a two-stage validated automated evaluation framework. It utilizes a deep learning model to extract key intermediate variables, such as emotion intensity, speech rate, and posture, from task videos and audio, using expert evaluation conclusions as the training benchmark. Subsequently, a multimodal fusion model based on attention mechanisms intelligently integrates these variables, and uses psychometric scale data for final training and validation. Ultimately, a detailed evaluation report is automatically generated. This training and validation mechanism, which incorporates both expert evaluation conclusions and psychometric data, not only significantly improves the accuracy and reliability of the entire evaluation system but also enables it to generate a comprehensive and insightful evaluation report, providing a solid scientific basis for team ability analysis and development.

[0048] Furthermore, such as Figure 3 As shown, based on the above-mentioned user collaboration and interaction capability assessment method based on multimodal data, the present invention also provides a user collaboration and interaction capability assessment system based on multimodal data, wherein the user collaboration and interaction capability assessment system based on multimodal data includes: The multimodal data acquisition module 51 is used to design team tasks under stress situations, select multiple users to participate in the team tasks, and acquire multimodal data of multiple users in the team tasks. The multimodal data includes facial expression data, voice data, and posture data. The multimodal feature extraction module 52 is used to extract features from the multimodal data to obtain multimodal intermediate variables, and input the multimodal intermediate variables into the trained multimodal intermediate variable analysis model for feature extraction to obtain the current multimodal features. The current multimodal features include the current emotional state, current emotional intensity, current leadership ability, current collaboration style, current posture focus, and current task participation. The multimodal feature fusion evaluation module 53 is used to input the current multimodal features into the trained multimodal fusion and evaluation model to perform multi-feature fusion and evaluation, and obtain the evaluation results of the collaboration and interaction capabilities of the multiple users.

[0049] Furthermore, such as Figure 4 As shown, based on the above-mentioned method and system for evaluating user collaboration and interaction capabilities based on multimodal data, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 4 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0050] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a user collaboration and interaction capability evaluation program 40 based on multimodal data. This multimodal data-based user collaboration and interaction capability evaluation program 40 can be executed by the processor 10, thereby implementing the multimodal data-based user collaboration and interaction capability evaluation method of this application.

[0051] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the user collaboration and interaction capability assessment method based on multimodal data.

[0052] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0053] In one embodiment, when the processor 10 executes the user collaboration and interaction capability assessment program 40 based on multimodal data in the memory 20, the following steps are performed: Design a team task under stress, select multiple users to participate in the team task, and obtain multimodal data of multiple users in the team task, including facial expression data, voice data and posture data; Feature extraction is performed on the multimodal data to obtain multimodal intermediate variables. The multimodal intermediate variables are then input into a trained multimodal intermediate variable analysis model for feature extraction to obtain current multimodal features. The current multimodal features include current emotional state, current emotional intensity, current leadership ability, current collaboration style, current posture focus, and current task participation. The current multimodal features are input into the trained multimodal fusion and evaluation model to perform multimodal feature fusion and evaluation, thereby obtaining the evaluation results of the collaboration and interaction capabilities of each of the users.

[0054] Specifically, the team task under design stress involves selecting multiple users to participate in the team task and acquiring multimodal data of the multiple users in the team task, including: Based on gamification theory and stressor theory, the team task was designed, and multiple users were selected to participate in the team task. Record all video and audio data in the team's tasks in real time, and collect multimodal data of each user from all the video and audio data.

[0055] The multimodal intermediate variables include: facial key points, confidence scores for specific emotions, and number of emotion category changes corresponding to the facial expression data; speech rate, pitch, fluctuation range, number of positive words, number of negative words, speaking order, interruption frequency, and speaking duration distribution corresponding to the speech data; and body orientation, gesture frequency, and posture type corresponding to the posture data.

[0056] Specifically, the step of extracting features from the multimodal data to obtain multimodal intermediate variables includes: The facial expression data was feature extracted using the MediaPipe facial model to obtain facial key points and confidence scores for specific emotions, and the number of changes in emotion categories was analyzed in conjunction with a specific time window. The speech data is digitally processed to quantify the user's speech rate, pitch, and fluctuation range. The speech data is then transcribed to obtain speech text. Natural language processing technology is used to perform sentiment analysis on the speech text to obtain the number of positive words and the number of negative words. The speech data is identified using the TalkNet model, and the speaking duration and speaking time distribution are quantified. The user's speaking order, interruption frequency, and speaking duration distribution are then extracted from the speaking duration and speaking time distribution. The posture data is analyzed frame by frame using the YOLO Pose model to obtain the position coordinates of multiple posture key points. Based on the position coordinates, the user's body orientation, gesture frequency, and posture type are obtained.

[0057] The step of inputting the multimodal intermediate variables into a trained multimodal intermediate variable analysis model for feature extraction to obtain the current multimodal features includes, prior to: Collect historical multimodal intermediate variables from historical team tasks, and obtain the multimodal features labeled by experts for the historical multimodal intermediate variables; A multimodal intermediate variable analysis model is constructed, using the historical multimodal intermediate variables as data samples and the multimodal features as labels; The multimodal intermediate variable analysis model is trained using the data samples and the labels to obtain a trained multimodal intermediate variable analysis model.

[0058] The multimodal features include emotional state, emotional intensity, leadership ability, collaborative style, postural focus, and task engagement. The emotional state and the emotional intensity are obtained by labeling intermediate variables corresponding to historical facial expression data; The leadership ability and the collaboration style are obtained by labeling intermediate variables corresponding to historical voice data; The posture focus and the task engagement are obtained by labeling intermediate variables corresponding to historical posture data.

[0059] The step of inputting the current multimodal features into a trained multimodal fusion and evaluation model to perform multi-feature fusion and evaluation, and obtaining evaluation results of the collaboration and interaction capabilities of multiple users, further includes: Historical multimodal intermediate variables are input into a trained multimodal intermediate variable analysis model for feature extraction to obtain historical multimodal features; A multimodal fusion and evaluation model based on an attention mechanism is constructed, the multimodal fusion and evaluation model including a multimodal feature fusion module and a multimodal feature evaluation module; The historical multimodal features are input into the multimodal fusion and evaluation model for vectorization to obtain vectorized features. The multimodal feature fusion module assigns weights to the vectorized features under different task area scenarios and performs attention-weighted fusion based on the assigned weights to obtain a fused feature vector. The user's collaboration and interaction abilities are assessed using a psychometric scale, and the assessed results are used as labels and the fused feature vector is used as a data sample. The data samples are input into the multimodal feature evaluation module for evaluation and prediction, and the predicted evaluation results of collaboration and interaction capabilities are output. Based on the predicted evaluation results and the labels, the multimodal fusion and evaluation model is trained and optimized to obtain a trained multimodal fusion and evaluation model.

[0060] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a user collaboration and interaction capability assessment program based on multimodal data, and the user collaboration and interaction capability assessment program based on multimodal data, when executed by a processor, implements the steps of the user collaboration and interaction capability assessment method based on multimodal data as described above.

[0061] In summary, this invention provides a method, system, and terminal for evaluating user collaboration and interaction capabilities based on multimodal data. The method includes: designing a team task under stress; selecting multiple users to participate in the team task; and acquiring multimodal data of the multiple users in the team task, including facial expression data, voice data, and posture data; extracting features from the multimodal data to obtain multimodal intermediate variables; inputting the multimodal intermediate variables into a trained multimodal intermediate variable analysis model for feature extraction to obtain current multimodal features, including current emotional state, current emotional intensity, current leadership ability, current collaboration style, current posture focus, and current task participation; and inputting the current multimodal features into a trained multimodal fusion and evaluation model for multi-feature fusion and evaluation to obtain the individual collaboration and interaction capability evaluation results for each of the multiple users. This invention constructs a two-stage validated automated evaluation framework, which greatly improves the accuracy and reliability of user collaboration and interaction capability evaluation through a training and validation mechanism that incorporates both expert evaluation conclusions and psychometric data.

[0062] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0063] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0064] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for evaluating user collaboration and interaction capabilities based on multimodal data, characterized in that, The method for evaluating user collaboration and interaction capabilities based on multimodal data includes: Design a team task under stress, select multiple users to participate in the team task, and obtain multimodal data of multiple users in the team task, including facial expression data, voice data and posture data; Feature extraction is performed on the multimodal data to obtain multimodal intermediate variables. The multimodal intermediate variables are then input into a trained multimodal intermediate variable analysis model for feature extraction to obtain current multimodal features. The current multimodal features include current emotional state, current emotional intensity, current leadership ability, current collaboration style, current posture focus, and current task participation. The current multimodal features are input into the trained multimodal fusion and evaluation model to perform multimodal feature fusion and evaluation, thereby obtaining the evaluation results of the collaboration and interaction capabilities of each of the users.

2. The method for evaluating user collaboration and interaction capabilities based on multimodal data according to claim 1, characterized in that, The team task under the design stress scenario involves selecting multiple users to participate in the team task and acquiring multimodal data of the multiple users in the team task, specifically including: Based on gamification theory and stressor theory, the team task was designed, and multiple users were selected to participate in the team task. Record all video and audio data in the team's tasks in real time, and collect multimodal data of each user from all the video and audio data.

3. The method for evaluating user collaboration and interaction capabilities based on multimodal data according to claim 1, characterized in that, The multimodal intermediate variables include: facial key points, confidence scores for specific emotions, and number of emotion category changes corresponding to the facial expression data; speech rate, pitch, fluctuation amplitude, number of positive words, number of negative words, speaking order, interruption frequency, and speaking duration distribution corresponding to the speech data; and body orientation, gesture frequency, and posture type corresponding to the posture data.

4. The method for evaluating user collaboration and interaction capabilities based on multimodal data according to claim 3, characterized in that, The step of extracting features from the multimodal data to obtain multimodal intermediate variables specifically includes: The facial expression data was feature extracted using the MediaPipe facial model to obtain facial key points and confidence scores for specific emotions, and the number of changes in emotion categories was analyzed in conjunction with a specific time window. The speech data is digitally processed to quantify the user's speech rate, pitch, and fluctuation range. The speech data is then transcribed to obtain speech text. Natural language processing technology is used to perform sentiment analysis on the speech text to obtain the number of positive words and the number of negative words. The speech data is identified using the TalkNet model, and the speaking duration and speaking time distribution are quantified. The user's speaking order, interruption frequency, and speaking duration distribution are then extracted from the speaking duration and speaking time distribution. The posture data is analyzed frame by frame using the YOLO Pose model to obtain the position coordinates of multiple posture key points. Based on the position coordinates, the user's body orientation, gesture frequency, and posture type are obtained.

5. The method for evaluating user collaboration and interaction capabilities based on multimodal data according to claim 1, characterized in that, The step of inputting the multimodal intermediate variables into a trained multimodal intermediate variable analysis model for feature extraction to obtain the current multimodal features also includes: Collect historical multimodal intermediate variables from historical team tasks, and obtain the multimodal features labeled by experts for the historical multimodal intermediate variables; A multimodal intermediate variable analysis model is constructed, using the historical multimodal intermediate variables as data samples and the multimodal features as labels; The multimodal intermediate variable analysis model is trained using the data samples and the labels to obtain a trained multimodal intermediate variable analysis model.

6. The method for evaluating user collaboration and interaction capabilities based on multimodal data according to claim 5, characterized in that, The multimodal features include emotional state, emotional intensity, leadership ability, collaborative style, postural focus, and task engagement. The emotional state and the emotional intensity are obtained by labeling intermediate variables corresponding to historical facial expression data; The leadership ability and the collaboration style are obtained by labeling intermediate variables corresponding to historical voice data; The posture focus and the task engagement are obtained by labeling intermediate variables corresponding to historical posture data.

7. The method for evaluating user collaboration and interaction capabilities based on multimodal data according to claim 1, characterized in that, The step of inputting the current multimodal features into a trained multimodal fusion and evaluation model to perform multimodal feature fusion and evaluation, and obtaining evaluation results of the collaboration and interaction capabilities of multiple users, also includes: Historical multimodal intermediate variables are input into a trained multimodal intermediate variable analysis model for feature extraction to obtain historical multimodal features; A multimodal fusion and evaluation model based on an attention mechanism is constructed, the multimodal fusion and evaluation model including a multimodal feature fusion module and a multimodal feature evaluation module; The historical multimodal features are input into the multimodal fusion and evaluation model for vectorization to obtain vectorized features. The multimodal feature fusion module assigns weights to the vectorized features under different task area scenarios and performs attention-weighted fusion based on the assigned weights to obtain a fused feature vector. The user's collaboration and interaction abilities are assessed using a psychometric scale, and the assessed results are used as labels and the fused feature vector is used as a data sample. The data samples are input into the multimodal feature evaluation module for evaluation and prediction, and the predicted evaluation results of collaboration and interaction capabilities are output. Based on the predicted evaluation results and the labels, the multimodal fusion and evaluation model is trained and optimized to obtain a trained multimodal fusion and evaluation model.

8. A user collaboration and interaction ability assessment system based on multimodal data, characterized in that, The user collaboration and interaction capability assessment system based on multimodal data includes: A multimodal data acquisition module is used to design team tasks under stress, select multiple users to participate in the team tasks, and acquire multimodal data of the multiple users in the team tasks. The multimodal data includes facial expression data, voice data, and posture data. The multimodal feature extraction module is used to extract features from the multimodal data to obtain multimodal intermediate variables. The multimodal intermediate variables are then input into a trained multimodal intermediate variable analysis model for feature extraction to obtain the current multimodal features. The current multimodal features include the current emotional state, current emotional intensity, current leadership ability, current collaboration style, current posture focus, and current task participation. The multimodal feature fusion and evaluation module is used to input the current multimodal features into the trained multimodal fusion and evaluation model to perform multi-feature fusion and evaluation, and obtain the evaluation results of the collaboration and interaction capabilities of each of the users.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a user collaboration and interaction capability assessment program based on multimodal data stored in the memory and executable on the processor. When the user collaboration and interaction capability assessment program based on multimodal data is executed by the processor, it implements the steps of the user collaboration and interaction capability assessment method based on multimodal data as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a user collaboration and interaction capability assessment program based on multimodal data, which, when executed by a processor, implements the steps of the user collaboration and interaction capability assessment method based on multimodal data as described in any one of claims 1-7.

Images