A method and system for evaluating team collaboration ability based on audio analysis

Abstract

The present invention discloses a team collaboration ability evaluation method and system based on audio analysis, which starts a team collaboration task; records the voice data generated during the team's task execution; performs denoising on the voice data and transcribes the human voice audio into text; pre-processes the transcribed text and extracts various task expression features in the text; inputs the extracted various task expression features into an established machine learning model or a deep neural network model for classification, and screens out positive expression features and negative expression features; obtains a team collaboration score Q based on the frequency of positive expression features and team task performance; and evaluates the team collaboration ability level based on the team collaboration score Q. The present invention naturally relaxes during the team's task execution without external interference, calculates the team collaboration score by extracting five aspects of task expression features, and accurately measures the level of team collaboration ability, and is applied to personnel training and team collaboration ability training.

Description

Technical Field

[0001] The present invention relates to the field of team competition and collaboration, and in particular to a team collaboration capability assessment method and system based on audio analysis. Background Art

[0002] In daily work, study and life, it is very important to cultivate teamwork ability. In situations involving cooperation and competition between teams, the tacit cooperation between team members can form a strong external competitiveness. On the contrary, if the cooperation ability between team members varies greatly, affecting the degree of tacit cooperation between members, the overall competitiveness of the team will be greatly reduced, which is not conducive to the improvement of team work quality and individual ability of members. Especially in certain industry fields, such as the medical field, in scenarios such as surgery, the ability of teamwork is even more important. Therefore, correct guidance and healthy planning of team building are crucial, and an important prerequisite for effective team capacity training is to effectively evaluate the teamwork ability of individuals and screen out individuals with relatively weak teamwork ability and stronger demand for training.

[0003] At present, the means of evaluating teamwork ability mainly include scales and observations in simulated situations or natural situations. Commonly used scales include the Clinical Team Assessment Scale (CTS) and the Nurse Teamwork Questionnaire. However, the scales mainly rely on the subjective reports of the subjects and are easily affected by factors such as social desirability, different individuals' understanding of the questions, and intentional concealment. In addition, the questionnaire scales ask about the individual's daily work status, their own beliefs and other related content, and lack relevant data on the individual's performance in real interactive situations. Observational evaluation in simulated situations or real situations has higher ecological validity and can better reflect the individual's cooperative behavior and cooperative ability in real or near-real work situations. However, this observational evaluation method requires the participation of two or more observers, and the observers need to be systematically trained before the evaluation begins. In addition, the definition of cooperative behavior in this evaluation method is relatively subjective, and the observers need to judge and define the behavior type based on pre-set standards. There is no method that can capture objective indicators in real teamwork, conduct automated analysis, and evaluate individual teamwork ability. Summary of the invention

[0004] In order to solve the above-mentioned technical problems and realize accurate and objective evaluation of team collaboration ability at low cost, the present invention provides a team collaboration ability evaluation method and system based on audio analysis.

[0005] The technical solutions adopted are as follows:

[0006] In one aspect, the present invention provides a method for evaluating teamwork ability based on audio analysis, the method comprising: starting a teamwork task;

[0007] Record the voice data generated during the team's mission execution;

[0008] De-noise the speech data and transcribe the human voice audio into text;

[0009] Preprocess the transcribed text to extract various task expression features in the text;

[0010] Input the extracted various task expression features into the established machine learning model or deep neural network model for classification, and screen out positive expression features and negative expression features;

[0011] Based on the frequency of positive expression characteristics and team task performance, the team collaboration score Q was obtained;

[0012] The team collaboration ability level is evaluated based on the team collaboration score Q.

[0013] The calculation formula of the teamwork score Q is:

[0014] Q = (0.5*(number of positive expression features / (number of positive expression features + number of negative expression features))+0.5*(number of successfully completed tasks / total number of tasks))*100;

[0015] The value of Q is a score between 0 and 100.

[0016] Furthermore, the team collaboration ability level is evaluated based on the team collaboration score Q, and its specific evaluation method is: setting team collaboration score thresholds a and b, when the obtained team collaboration score Q satisfies b>Q>a, the team collaboration ability level is evaluated as medium; when the obtained team collaboration score Q satisfies Q≥b, the team collaboration ability level is evaluated as high; when the obtained team collaboration score Q satisfies Q≤a, the team collaboration ability level is evaluated as low.

[0017] Preferably, the task expression features include: task relevance, cohesion, sentiment index, expression of approval and expression of confusion, or a combination of several of them.

[0018] Furthermore, the method for preprocessing the transcribed text and extracting the task expression features in the text is: preprocessing the transcribed text by using corpus cleaning, word segmentation, part-of-speech tagging and stop word removal methods, and then using an open source natural language processing toolkit to perform spelling check, word segmentation and part-of-speech tagging on the preprocessed text; combining the established task-related corpus, approval expression corpus and confusion expression corpus, calculating the consistency of the sentence with the corpus in the corresponding corpus, and obtaining the task-related feature value, approval expression feature value and confusion expression feature value respectively; at the same time, calculating the similarity between the sentence and the previous sentence to obtain the cohesion feature value, calculating the sentiment index of the sentence to obtain the sentiment index feature value.

[0019] Preferably, the team collaboration task adopted is a two-player cooperative bomb disposal game, and the two-player cooperative bomb disposal game has a built-in microphone or on-site microphone recording.

[0020] On the other hand, the present invention also provides a team collaboration capability assessment system based on audio analysis, the system comprising: a team collaboration task module for presenting the content of tasks completed by team members in collaboration;

[0021] An audio recording module is used to record the voice data generated during the collaborative task completion process;

[0022] The audio preprocessing module is used to denoise the recorded voice data and extract the human voice;

[0023] Audio-to-text module, used to transcribe human voice audio into text;

[0024] A text preprocessing module, used to preprocess the transcribed text;

[0025] The feature extraction module is used to extract features from the preprocessed text, and extract task-related features, cohesion features, sentiment index features, approval expression features, and confusion expression features from the text;

[0026] A feature classification module inputs the extracted features into the established machine learning model or deep neural network model to classify them into positive expressions and negative expressions;

[0027] The calculation module calculates the team collaboration score based on the frequency of positive expressions in the feature classification module and the team task performance;

[0028] The evaluation module makes an overall evaluation of the team collaboration ability based on the obtained team collaboration score and the set score threshold.

[0029] Furthermore, the feature extraction module also includes: a task content-related expression corpus, an approval expression corpus and a confusion expression corpus. The text processed by the text preprocessing module is respectively compared with the contents of the task content-related expression corpus, the approval expression corpus and the confusion expression corpus to extract the corresponding relevant task feature values; the feature extraction module obtains the cohesion feature value by calculating the similarity between the sentence and the previous sentence, and calculates the emotion index of the sentence to obtain the emotion index feature value.

[0030] The technical solution of the present invention has the following advantages:

[0031] A. The present invention obtains audio data of the team when performing collaborative tasks, converts the audio into text, extracts task-related features generated in the collaborative process, classifies the extracted features through machine learning methods, and obtains a team collaboration score based on the ratio of the number of positive expression features to the sum of the number of positive expressions and negative expressions and the team task performance. The team collaboration ability is evaluated based on the score. The entire collaborative communication process is naturally relaxed and not disturbed by the external environment. The team collaboration ability can be accurately measured through the collaboration score.

[0032] B. The present invention extracts features from five aspects (task relevance, cohesion, emotional index, expression of approval and expression of confusion), and extracts the cooperation status in the collaboration process based on the positive and negative expressions of the feature expressions. The present invention can accurately know the coordination between members in the interaction process. The obtained data is closer to the real state of collaboration, has the advantages of high efficiency and low cost, and can be applied to personnel training and team collaboration ability training. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the specific embodiments of the present invention, the drawings required for use in the specific embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 It is a flow chart of the discrimination method provided by the present invention;

[0035] Figure 2 It is a specific flow chart of teamwork ability assessment provided by the present invention;

[0036] Figure 3 This is a composition diagram of the teamwork capability evaluation system provided by the present invention. DETAILED DESCRIPTION

[0037] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] like Figure 1 and Figure 2 As shown, the present invention provides a team collaboration ability assessment method based on audio analysis, comprising the following steps:

[0039] Step 1, start the team collaboration task; the team collaboration task used here includes a two-player cooperative bomb disposal game. Of course, other tasks involving multi-player collaboration can also be used. For example, for a bomb disposal game involving multi-player collaboration, the game tasks can be broken down, with half of the people responsible for delivering instructions and the other half responsible for operations, and then different types of tasks are assigned to different people to complete.

[0040] Step 2, record the voice data generated during the team's task execution;

[0041] Step 3: De-noise the voice data and transcribe the human voice audio into text; mainly remove background noise and other noise, extract the communication sound between collaborative team members, and filter and remove the background sound through relevant processing software.

[0042] Step 4: preprocess the transcribed text to extract various task expression features in the text.

[0043] The various task expression features here specifically involve five aspects, of course, but are not limited to the five aspects provided. The five aspects of expression features include: task relevance, cohesion, emotional index, approval expression and confusion expression, one or a combination of several.

[0044] Data preprocessing:

[0045] After the sound is converted into text, the text is preprocessed using methods such as corpus cleaning, word segmentation, part-of-speech tagging, and delistening. Then, the open source Natural Language Processing (NLP) toolkit is used to perform spelling check, word segmentation, part-of-speech tagging, and subsequent feature extraction.

[0046] The feature extraction method is as follows:

[0047] For each sentence, the following five types of features are extracted: task relevance, cohesion, sentiment index, expression of approval, and expression of confusion. For the three features of task relevance, expression of approval, and expression of confusion, corresponding corpora are established respectively, and the consistency between the sentence and the corpus is calculated, which is a value between 0 and 1. When the sentence and the corpus are highly consistent, the value is 1, and the value is 0 if it is completely inconsistent. The feature value of 0-1 is given according to the expression of the sentence. The corpus example is as follows:

[0048] Task-related features: such as "cut the red wire", "press the button", "bomb number", etc.

[0049] Characteristics of expressions of agreement: such as "You are right", "I agree", "Let's do it this way", etc.

[0050] Confused expression characteristics: such as "what to do", "I don't know", "what to do now", etc.

[0051] Cohesion feature: Calculates the similarity between a sentence and the previous sentence, which is a value between 0-1. If the previous and next sentences are completely unrelated, the value is 1; if the previous and next sentences are highly related, the value is 0. The value is between 0-1 depending on the degree of correlation between the previous and next sentences.

[0052] Sentiment Index Feature: Calculates the sentiment index of a sentence, with values ​​between -1 (negative) and +1 (positive).

[0053] Step 5, input the extracted various task expression feature values ​​into the machine learning model or deep neural network model for classification, and divide them into two types: positive expression and negative expression. That is, for each sentence, the values ​​of the five features of task relevance, cohesion, emotional index, expression of approval, and expression of confusion are input into the support vector machine (SVM) machine learning model. In the model training stage, the data is manually labeled with two types of expressions, and then the data is divided into a training set and a test set. The model is trained with the training set, and then the accuracy of the model is tested with the test set. The parameters of the model are further adjusted by grid optimization and other methods to obtain a trained machine learning model that can be used for classification. Classification is performed through the established machine learning model or deep neural network model, and then two types of expression features are obtained, and the number of expression feature entries of each type is counted for the next step of calculation.

[0054] Step 6, based on the frequency of positive expression characteristics and team task performance, the team collaboration score Q is obtained.

[0055] The calculation formula for teamwork score Q is:

[0056] Q = (0.5*(number of positive expression features / (number of positive expression features+number of negative expression features))+0.5*(number of successfully completed tasks / total number of tasks))*100, where the value of Q is a score between 0-100).

[0057] Step 7: Evaluate the team collaboration ability level based on the team collaboration score Q.

[0058] The team collaboration score thresholds a and b are set. When the obtained team collaboration score b>Q>a, the team collaboration ability level is medium; when Q≥b, the team collaboration ability level is high; when Q≤a, the team collaboration ability level is low.

[0059] The score thresholds used here are preferably a=30, b=70. When the obtained score is ≥70, the team collaboration ability level is evaluated as high. When the obtained score is ≤30, the team collaboration ability level is evaluated as low. When the obtained score is between 30 and 70, the team collaboration ability level is evaluated as medium.

[0060] like Figure 3 As shown, on the other hand, the present invention also provides a team collaboration capability assessment system based on audio analysis, including: a team collaboration task module, an audio recording module, an audio preprocessing module, an audio conversion text module, a text preprocessing module, a feature extraction module, a feature classification module, a calculation module and an evaluation module; wherein the team collaboration task module is used to present the task content completed by team members in cooperation, and the team collaboration task adopted is preferably a two-player cooperative bomb disposal game; the audio recording module is used to record the voice data generated in the process of collaborative task completion, and the two-player cooperative bomb disposal game can be recorded using a built-in microphone or an on-site microphone. The audio preprocessing module is used to denoise the recorded voice data and extract the human voice; the audio-to-text module is used to transcribe the human voice audio into text; the text preprocessing module is used to preprocess the transcribed text; the feature extraction module is used to extract features from the preprocessed text, extracting task-related features, cohesion features, emotional index features, approval expression features and confusion expression features in the text; the feature classification module is used to classify the extracted features into positive expressions and negative expressions; the calculation module calculates the team collaboration score based on the feature classification in the classification module; the evaluation module makes an overall evaluation of the team collaboration ability based on the obtained team collaboration score and the set score threshold.

[0061] The feature extraction module also includes: task content related expression corpus, approval expression corpus and confusion expression corpus. The text processed by the text preprocessing module is compared with the contents of each task content related expression corpus, approval expression corpus and confusion expression corpus respectively to extract the corresponding relevant task features. The task content related expression corpus contains standard task content related expression sentences, such as "cut the red line" and "press the button", etc., and the extracted sentences are compared to see whether they contain standard sentences in the corpus. The approval expression corpus contains standard approval expression sentences, such as "you are right", "I agree", etc., and the extracted sentences contain such expression sentences; the confusion expression corpus contains standard sentences, such as "what should we do now", "I don't know", etc., and the extracted sentences are compared to see whether they contain such standard sentences.

[0062] Cohesion features are used to determine the consistency between two consecutive sentences.

[0063] The sentiment index feature is the sentiment index of the discourse, with values ​​ranging from -1 (negative) to +1 (positive).

[0064] The feature extraction module calculates the similarity between a sentence and the previous sentence to obtain the cohesion feature value, and calculates the sentiment index of the sentence to obtain the sentiment index feature value. The feature classification module inputs the extracted feature value into the established machine learning model or deep neural network model, and classifies it according to positive expression and negative expression. Then, according to the classification situation, the calculation module calculates and the evaluation module evaluates it to obtain the level of team collaboration ability.

[0065] Anything not described in the present invention is applicable to the prior art.

[0066] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. However, the obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A method for evaluating teamwork ability based on audio analysis, characterized in that: The method is as follows: starting a team collaboration task; Record the voice data generated during the team's mission execution; De-noise the speech data and transcribe the human voice audio into text; Preprocess the transcribed text to extract various task expression features in the text, including: task relevance, cohesion, sentiment index, approval expression and confusion expression, or a combination of several of them; Input the extracted various task expression features into the established machine learning model or deep neural network model for classification, and screen out positive expression features and negative expression features; According to the frequency of positive expression characteristics and team task performance, the team collaboration score Q is obtained; the calculation formula of the team collaboration score Q is: Q = (0.5*(number of positive expression features / (number of positive expression features + number of negative expression features))+0.5*(number of successfully completed tasks / total number of tasks))*100, where the value of Q is a score between 0 and 100; The team collaboration ability level is evaluated based on the team collaboration score Q.

2. The method for evaluating teamwork ability based on audio analysis according to claim 1, characterized in that: The specific evaluation method of evaluating the team collaboration ability level based on the team collaboration score Q is: setting team collaboration score thresholds a and b, when the obtained team collaboration score Q satisfies b>Q>a, the team collaboration ability level is evaluated as medium; when the obtained team collaboration score Q satisfies Q≥b, the team collaboration ability level is evaluated as high; when the obtained team collaboration score Q satisfies Q≤a, the team collaboration ability level is evaluated as low.

3. The method for evaluating teamwork ability based on audio analysis according to claim 1, characterized in that: The method of preprocessing the transcribed text and extracting the task expression features in the text is as follows: preprocessing the transcribed text by using corpus cleaning, word segmentation, part-of-speech tagging and stop word removal methods, and then using an open source natural language processing toolkit to perform spelling check, word segmentation and part-of-speech tagging on the preprocessed text; combining the established task-related corpus, approval expression corpus and confusion expression corpus, calculating the consistency of the sentence with the corpus in the corresponding corpus, and obtaining the task-related feature value, the approval expression feature value and the confusion expression feature value respectively; at the same time, calculating the similarity between the sentence and the previous sentence to obtain the cohesion feature value, calculating the sentiment index of the sentence to obtain the sentiment index feature value.

4. The method for evaluating teamwork ability based on audio analysis according to claim 1, characterized in that: The team collaboration task adopted is a two-player cooperative bomb disposal game, and the two-player cooperative bomb disposal game has a built-in microphone or on-site microphone recording.

5. A team collaboration ability assessment system based on audio analysis, characterized in that: The system includes: a team collaboration task module for presenting the content of tasks completed by team members in collaboration; An audio recording module is used to record the voice data generated during the collaborative task completion process; The audio preprocessing module is used to denoise the recorded voice data and extract the human voice; Audio-to-text module, used to transcribe human voice audio into text; A text preprocessing module, used to preprocess the transcribed text; The feature extraction module is used to extract features from the preprocessed text, and extract task-related features, cohesion features, sentiment index features, approval expression features, and confusion expression features from the text; A feature classification module inputs the extracted features into the established machine learning model or deep neural network model to classify them into positive expressions and negative expressions; The calculation module calculates the team collaboration score based on the frequency of positive expressions in the feature classification module and the team task performance; The evaluation module makes an overall evaluation of the team collaboration ability based on the obtained team collaboration score and the set score threshold.

6. The team collaboration ability assessment system based on audio analysis according to claim 5 is characterized in that: The feature extraction module also includes: a task content-related expression corpus, an approval expression corpus and a confusion expression corpus. The text processed by the text preprocessing module is respectively compared with the contents of the task content-related expression corpus, the approval expression corpus and the confusion expression corpus to extract the corresponding relevant task feature values; the feature extraction module obtains the cohesion feature value by calculating the similarity between the sentence and the previous sentence, and calculates the emotion index of the sentence to obtain the emotion index feature value.

Images