Man-machine interaction system efficiency evaluation method and system based on multi-modal data

Abstract

The invention provides a man-machine interaction system efficiency evaluation method and system based on multi-modal data, and belongs to the field of man-machine interaction, and the method comprises the steps: carrying out the function support degree analysis of each function dimension, and obtaining a function diversity evaluation through combining a relative weight value; calculating response delay corresponding to each function dimension, and obtaining agility evaluation in combination with a weight coefficient of a response delay index; calculating an accuracy rate corresponding to each function dimension, and obtaining an accuracy evaluation in combination with a weight coefficient of an accuracy rate index; and based on functional diversity evaluation, agility evaluation and accuracy evaluation, in combination with the corresponding evaluation weight coefficients, performing overall evaluation on the efficiency of the man-machine interaction system based on the multi-modal data. According to the method, the efficiency of the man-machine interaction system can be comprehensively analyzed.

Description

Technical Field

[0001] The present application belongs to the field of human-computer interaction, and more specifically, to a method and system for evaluating the effectiveness of a human-computer interaction system based on multimodal data. Background Art

[0002] In the operating system, human-computer interaction technology plays a vital role, which is directly related to the communication efficiency between users and computer systems and user experience.

[0003] With the development of emerging technologies such as sensor technology, intelligent computing technology, cloud computing and big data, human-computer interaction technology has gradually become one of the major topics in the information field, and is constantly developing in the direction of people-oriented, enhanced physical sensation and user convenience. The traditional mouse, keyboard and light interaction methods can no longer meet the needs of new, efficient and convenient human-computer interaction. The natural and efficient interaction between people and systems and the accurate understanding of user intentions need to be solved. The rise of human-computer interaction technology based on multimodal data has further improved the efficiency and friendliness of human-computer interaction. It integrates the user's voice, expression, posture, gesture, text and handwriting information to realize natural language human-computer interaction, phrase interaction, body language and expression interaction, etc., and when there is ambiguity or ambiguity in the interaction, the system will also require clarification or confirmation.

[0004] At present, the more mature interaction methods include voice recognition interaction, gesture control, tactile interaction, blink detection and eye tracking technology, face detection and tracking, etc. With the dynamic integration and continuous development of information, a single mode of human-computer interaction no longer meets the human-centered natural, harmonious and efficient human-computer interaction needs, and more and more researchers are choosing a multi-modal fusion approach.

[0005] The current development focus of human-computer interaction technology is mainly on multimodal data feature extraction, data fusion and user intention inference. The technical evaluation of human-computer interaction systems mainly focuses on the accuracy of human-computer interaction intention prediction in a single aspect such as gestures, voice or eye movements, and it is impossible to make a comprehensive and accurate evaluation of the effectiveness of human-computer interaction systems. Summary of the invention

[0006] In view of the defects of the prior art, the purpose of this application is to provide a method and system for evaluating the performance of a human-computer interaction system based on multimodal data, aiming to solve the problems of incompleteness and poor accuracy in the technical evaluation of current human-computer interaction systems.

[0007] To achieve the above objectives, in a first aspect, the present application provides a method for evaluating the effectiveness of a human-computer interaction system based on multimodal data, comprising the following steps:

[0008] Step S1: Perform function support analysis on each function dimension, and obtain the function diversity evaluation of the human-computer interaction system by combining the relative weight value of each function dimension;

[0009] At the same time, the response delay corresponding to each functional dimension of the human-computer interaction system is calculated, and the agility evaluation of the human-computer interaction system is obtained by combining the weight coefficients of various response delay indicators;

[0010] At the same time, the accuracy of each functional dimension of the human-computer interaction system is calculated, and the accuracy evaluation of the human-computer interaction system is obtained by combining the weight coefficients of various accuracy indicators;

[0011] Step S2: Based on the functional diversity evaluation, agility evaluation and accuracy evaluation of the human-computer interaction system, combined with the evaluation weight coefficients corresponding to the functional diversity, agility and accuracy, an overall evaluation of the effectiveness of the human-computer interaction system based on multimodal data is performed;

[0012] Among them, each functional dimension includes at least eye tracking function, gesture recognition function, voice recognition function, electromyography acquisition function and environmental perception function.

[0013] Further preferably, the method for obtaining the relative weight value, the weight coefficient of the response delay index, and the weight coefficient of the accuracy index specifically comprises the following steps:

[0014] Determine the number n of indicator factors of each functional dimension, compare the importance of each two functional support, response delay and accuracy among the n indicator factors, and construct a weight matrix corresponding to the functional support, response delay and accuracy;

[0015] The weight matrices corresponding to the function support, response delay, and accuracy are normalized by column, and then added by row to obtain the added weight matrices corresponding to the function support, response delay, and accuracy;

[0016] The added matrices corresponding to function support, response delay and accuracy are normalized respectively to obtain the relative weight value of each function dimension, the weight coefficient of the response delay index, and the weight coefficient of the accuracy index.

[0017] Further preferably, the indicator factors of the eye tracking function include the ability to acquire gaze time, the ability to detect line of sight movement distance, the ability to detect pupil changes and the ability to acquire residence time; the indicator factors of the gesture recognition function include the ability to acquire finger movement path, the ability to detect movement speed, the ability to acquire gesture shape and the ability to acquire duration; the indicator factors of the speech recognition function include the ability to acquire timbre, the ability to acquire pitch and the ability to segment speech; the indicator factors of the electromyography acquisition function include the ability to acquire electromyography signals, the ability to acquire nerve conduction velocity and the ability to acquire muscle dynamic characteristics; the indicator factors of the environmental perception function include the ability to perceive temperature, the ability to perceive sound, the ability to perceive light and the ability to recognize objects.

[0018] Further preferably, the method for obtaining the function support of each function dimension is:

[0019] Determine whether the human-computer interaction system has the indicator factors of each functional dimension. If so, the support coefficient of the corresponding indicator factor is 1, otherwise it is 0; sum the support coefficients corresponding to the indicator factors in each functional dimension to obtain the functional support of each functional dimension.

[0020] Further preferably, step S2 specifically includes the following steps:

[0021] Compare the evaluation results of each of the functional diversity, agility and accuracy to obtain the relative importance index, construct the evaluation weight vector Weight=(weight1,weight2,weight3) of functional diversity, agility and accuracy, and combine the evaluation result vector F=(F1,F2,F3) of the functional diversity evaluation, agility evaluation and accuracy evaluation of the human-computer interaction system. T , to evaluate the effectiveness of the human-computer interaction system

[0022] Among them, weight1 is the evaluation weight coefficient of functional diversity; weight2 is the evaluation weight coefficient of agility; weight3 is the evaluation weight coefficient of accuracy; F1 is the evaluation result of functional diversity; F2 is the evaluation result of agility; F3 is the evaluation result of accuracy; S j is the functional support corresponding to functional dimension j, w j is the relative weight value corresponding to functional dimension j; delay j is the response delay index corresponding to functional dimension j, w' j is the weight coefficient of the response delay index corresponding to the functional dimension j; accu j is the accuracy index corresponding to functional dimension j, w” jis the weight coefficient of the accuracy index corresponding to functional dimension j; n is the total number of functional dimensions.

[0023] In a second aspect, the present application provides a human-computer interaction system effectiveness evaluation system based on multimodal data, comprising:

[0024] Functional diversity evaluation module, used to analyze the functional support of each functional dimension, and obtain the functional diversity evaluation of the human-computer interaction system by combining the relative weight value of each functional dimension;

[0025] The agility evaluation module is used to calculate the response delay corresponding to each functional dimension of the human-computer interaction system, and obtain the agility evaluation of the human-computer interaction system by combining the weight coefficients of various response delay indicators;

[0026] The accuracy evaluation module is used to calculate the accuracy corresponding to the functional dimension of the human-computer interaction system, and obtain the accuracy evaluation of the human-computer interaction system by combining the weight coefficients of various accuracy indicators;

[0027] The overall evaluation module is used to evaluate the functional diversity, agility and accuracy of the human-computer interaction system. It combines the evaluation weight coefficients corresponding to functional diversity, agility and accuracy to make an overall evaluation of the effectiveness of the human-computer interaction system based on multimodal data.

[0028] Among them, each functional dimension includes at least eye tracking function, gesture recognition function, voice recognition function, electromyography acquisition function and environmental perception function.

[0029] Further preferably, the functional diversity evaluation module includes a first weight acquisition unit, which is used to determine the number n of indicator factors of each functional dimension, compare the importance of every two functional supports in the n indicator factors, and construct a weight matrix corresponding to the functional support; normalize the weight matrix corresponding to the functional support by column, and then add them by row to obtain the added weight matrix corresponding to the functional support; normalize the added matrices corresponding to the functional support to obtain the relative weight values ​​of each functional dimension;

[0030] The agility evaluation module includes a second weight acquisition unit, which is used to determine the number n of indicator factors of each functional dimension, compare the importance of every two response delays in the n indicator factors, and construct a weight matrix corresponding to the response delay; normalize the weight matrix corresponding to the response delay by column, and then add them by row to obtain a weight matrix after the response delays are added; normalize the weight matrix after the response delays are added to obtain the weight coefficient of the response delay indicator;

[0031] The accuracy evaluation module includes a third weight acquisition unit, which is used to determine the number n of indicator factors of each functional dimension, compare the importance of every two accuracies of the n indicator factors, and construct a weight matrix corresponding to the accuracy; normalize the weight matrix corresponding to the accuracy by column, and then add them by row to obtain the weight matrix after the accuracy is added; normalize the weight matrix after the accuracy is added to obtain the weight coefficient of the accuracy indicator.

[0032] Further preferably, the indicator factors of the eye tracking function include the ability to acquire gaze time, the ability to detect line of sight movement distance, the ability to detect pupil changes and the ability to acquire residence time; the indicator factors of the gesture recognition function include the ability to acquire finger movement path, the ability to detect movement speed, the ability to acquire gesture shape and the ability to acquire duration; the indicator factors of the speech recognition function include the ability to acquire timbre, the ability to acquire pitch and the ability to segment speech; the indicator factors of the electromyography acquisition function include the ability to acquire electromyography signals, the ability to acquire nerve conduction velocity and the ability to acquire muscle dynamic characteristics; the indicator factors of the environmental perception function include the ability to perceive temperature, the ability to perceive sound, the ability to perceive light and the ability to recognize objects.

[0033] Further preferably, the method for acquiring the functional support of each functional dimension in the first weight acquisition unit is: determine whether the human-computer interaction system has the indicator factors of each functional dimension, if so, the support coefficient of the corresponding indicator factor is 1, otherwise it is 0; sum the support coefficients corresponding to the indicator factors in each functional dimension to obtain the functional support of each functional dimension.

[0034] Further preferably, the overall evaluation function of the human-computer interaction system effectiveness in the overall evaluation module is:

[0035] Among them, weight1 is the evaluation weight coefficient of functional diversity; weight2 is the evaluation weight coefficient of agility; weight3 is the evaluation weight coefficient of accuracy; F1 is the evaluation result of functional diversity; F2 is the evaluation result of agility; F3 is the evaluation result of accuracy; S j is the functional support corresponding to functional dimension j, w j is the relative weight value corresponding to functional dimension j; delay j is the response delay index corresponding to functional dimension j, w' j is the weight coefficient of the response delay index corresponding to the functional dimension j; accu j is the accuracy index corresponding to functional dimension j, w” j is the weight coefficient of the accuracy index corresponding to functional dimension j; n is the total number of functional dimensions.

[0036] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the programs stored in the memory are executed, the processor is used to execute the method described in the first aspect or a further preferred implementation of the first aspect.

[0037] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or a further preferred implementation of the first aspect.

[0038] In a fifth aspect, the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method described in the first aspect or a further preferred implementation of the first aspect.

[0039] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0040] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art:

[0041] The present application provides a method for evaluating the performance of a human-computer interaction system based on multimodal data. Compared with the traditional single performance evaluation method, the present application constructs a human-computer interaction system performance evaluation model from multiple dimensions such as functional diversity, agility and accuracy, and comprehensively considers various functional dimensions, including eye tracking function, gesture recognition function, speech recognition function, electromyography acquisition function and environmental perception function, and can comprehensively analyze the performance of the human-computer interaction system.

[0042] The present application provides a method for evaluating the performance of a human-computer interaction system based on multimodal data. For different application scenarios of the human-computer interaction system, the relative weight value, the weight coefficient of the response delay index, and the weight coefficient of the accuracy index are calculated. When evaluating the system performance, the evaluation weights corresponding to functional diversity, agility, and accuracy are also considered, thereby providing a more accurate evaluation of the performance of the human-computer interaction system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a data flow diagram of a human-computer interaction system effectiveness evaluation method based on multimodal data provided in an embodiment of the present application;

[0044] Figure 2 is a flowchart of the effectiveness evaluation of a human-computer interaction system based on multimodal data provided by an embodiment of the present application;

[0045] Figure 3It is a flow chart for evaluating the functional diversity of a human-computer interaction system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] The terms “first”, “second” and the like in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects.

[0048] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0049] In the description of the embodiments of the present application, unless otherwise specified, “plurality” means two or more than two.

[0050] The technical solutions provided in the embodiments of the present application are introduced.

[0051] In response to the lack of a comprehensive and systematic evaluation method for human-computer interaction systems, this application proposes a human-computer interaction system performance evaluation method based on multimodal data, aiming to comprehensively evaluate the performance of human-computer interaction systems from multiple dimensions such as diversity, accuracy, and agility, based on eye movement, gesture, voice, posture, and environmental multimodal data. This application provides a human-computer interaction system performance evaluation method based on multimodal data, including the following steps:

[0052] Step 1: Evaluate the functional diversity of the human-computer interaction system:

[0053] Establish a function list, analyze the function support of eye tracking, gesture recognition, voice recognition, electromyography, and environmental perception, and conduct a comprehensive analysis of the functional diversity of the human-computer interaction system based on the relative weights of each functional dimension;

[0054] Step 2: Evaluate the agility of the human-computer interaction system:

[0055] The eye tracking response delay, gesture recognition response delay, speech recognition response delay, electromyography acquisition response delay and environmental perception response delay of the human-computer interaction system are tested respectively; according to the characteristics of the system, the relative importance of each of the above response delay indicators is set, and the weight coefficient of each of the above response delay indicators is calculated to evaluate the agility of the human-computer interaction system;

[0056] Step 3: Evaluate the accuracy of the human-computer interaction system:

[0057] Through multiple tests, the accuracy of eye tracking, gesture recognition, speech recognition, electromyographic signal acquisition and environmental perception of the human-computer interaction system are calculated respectively; according to the characteristics of the system, the relative importance of each of the above accuracy indicators is set, and the weight coefficient of each of the above accuracy indicators is calculated to evaluate the accuracy of the human-computer interaction system;

[0058] Step 4: Combine the multi-dimensional evaluation results such as the system functional diversity evaluation results, system agility evaluation results, and system accuracy evaluation results calculated in steps 1, 2, and 3, as well as the weight coefficients corresponding to functional diversity, agility, and accuracy, to comprehensively evaluate and analyze the effectiveness of the human-computer interaction system based on multimodal data.

[0059] More specifically, step 1 is as follows:

[0060] The function list includes eye tracking, gesture recognition, voice recognition, electromyography acquisition and environmental perception. Each functional dimension uses corresponding indicator factors to evaluate the system's support for the function, as follows:

[0061] Analyze whether the human-computer interaction system has the ability to obtain eye gaze time, realize movement distance detection, pupil change detection, and retention time acquisition capabilities; combine the support capabilities and weight coefficients of the above capabilities to evaluate the eye tracking support of the human-computer interaction system;

[0062] Analyze whether the human-computer interaction system has the ability to obtain finger movement path, movement speed detection, gesture shape and duration, and evaluate the gesture recognition support of the human-computer interaction system based on the support capabilities and weight coefficients of the above capabilities;

[0063] Analyze whether the human-computer interaction system has the ability to obtain timbre, pitch, and speech segmentation, and evaluate the speech recognition support of the human-computer interaction system based on the support capabilities and weight coefficients of the above capabilities;

[0064] Analyze whether the human-computer interaction system has the ability to acquire electromyographic signals, nerve conduction velocity, and muscle dynamic characteristics; evaluate the electromyographic acquisition support of the human-computer interaction system based on the support capabilities and weight coefficients of the above capabilities;

[0065] Analyze whether the human-computer interaction system has the ability to perceive temperature, sound, light, and objects. Combine the support capabilities and weight coefficients of the above capabilities to evaluate the environmental perception support of the human-computer interaction system.

[0066] Based on the analyzed support for eye tracking, gesture recognition, speech recognition, electromyography acquisition and environmental perception, combined with the relative weight coefficients of each functional dimension, a comprehensive evaluation of the functional diversity of the human-computer interaction system is conducted.

[0067] Example 1

[0068] Combination Figure 1 , Figure 2 and Figure 3 The human-computer interaction system performance evaluation method provided by the present application is described using an embodiment, which specifically includes the following steps:

[0069] The first step is to set the relative importance of each functional dimension in the human-computer interaction system and calculate the weight index of each functional dimension; more specifically, the following steps are included:

[0070] Step 1.1: Establish a relative importance scale, compare the importance of the first factor and the second factor, and determine the importance ratio p of the first factor relative to the second factor;

[0071] Step 1.2: Determine the number of indicator factors n for each functional dimension, compare the importance of every two of the n indicator factors, and record the importance ratio of factor a to factor b as p ab ;

[0072] Step 1.3: Construct the weight matrix

[0073] Step 1.4: Normalize the matrix P by column, and then add by row to get the matrix

[0074] Step 1.5: Finally, normalize the matrix Q to obtain the weight vector W, where

[0075] Step 2: Determine the support level of the human-computer interaction system for eye tracking, gesture recognition, speech recognition, electromyography acquisition, and environmental perception. The specific steps include:

[0076] Step 2.1: Establish a list of eye tracking support functions, including the ability to obtain gaze time, the ability to detect gaze movement distance, the ability to detect pupil changes, and the ability to obtain retention time; according to the list of eye tracking support functions, for function i, if the human-computer interaction system supports this function, the support coefficient of this function is e i is 1, otherwise e i 0; calculate eye tracking support Where n is the number of functions in the function list, e i is the support coefficient of function i;

[0077] Step 2.2: Establish a list of gesture recognition support functions, including the ability to obtain finger movement path, movement speed detection, gesture shape, and duration; according to the list of gesture recognition support functions, for function i, if the human-computer interaction system supports the function, record the support coefficient h of the function i is 1, otherwise h i 0; calculate gesture recognition support Where n is the number of features in the feature list, h i is the support coefficient of function i;

[0078] Step 2.3: Create a list of speech recognition support functions, including timbre acquisition capability, pitch acquisition capability, and speech segmentation capability; according to the list of speech recognition support functions, for function i, if the human-computer interaction system supports the function, record the support coefficient v of the function i is 1, otherwise v i 0; calculate the support of speech recognition Where n is the number of features in the feature list, v i is the support coefficient of function i;

[0079] Step 2.4: Establish a list of myoelectric acquisition support functions, including the ability to acquire myoelectric signals, the ability to acquire nerve conduction velocity, and the ability to acquire muscle dynamic characteristics; according to the list of myoelectric acquisition support functions, for function i, if the human-computer interaction system supports this function, then record the support coefficient m of this function i is 1, otherwise m i 0; calculate the support of electromyography acquisition Where n is the number of functions in the function list, m i is the support coefficient of function i;

[0080] Step 2.5: Establish a list of environmental perception support functions, including temperature perception, sound perception, light perception, and object recognition capabilities; according to the list of environmental perception support functions, for function i, if the human-computer interaction system supports this function, record the support coefficient t of this function i is 1, otherwise t i0; calculate the support of environmental perception Where n is the number of features in the feature list, t i is the support coefficient of function i.

[0081] Step 3: Based on the support of the human-computer interaction system for eye tracking, gesture recognition, voice recognition, electromyography acquisition and environmental perception, combined with the weight indicators of each type of function, a comprehensive evaluation of the functional diversity of the human-computer interaction system is conducted;

[0082] The specific steps include:

[0083] Step 3.1: Compare the support of each two functions of eye tracking, gesture recognition, speech recognition, electromyography acquisition and environmental perception to obtain the relative importance index, and use the method in the first step to determine the relative weight vector W = (w1, w2, w3, w4, w5);

[0084] Step 3.2: Construct the function support vector S = (S1, S2, S3, S4, S5) based on the calculation results in the second step T ;

[0085] Step 3.3: Based on the weight vector in step 3.1 and the function support vector in step 3.2, the functional diversity of the human-computer interaction system is evaluated. The evaluation result calculation formula is: Among them, S j is the support value corresponding to functional dimension j, w j is the relative weight value corresponding to functional dimension j. If only eye tracking, gesture recognition, speech recognition, electromyography acquisition and environmental perception functions are considered, then n = 5;

[0086] Step 4: Evaluate the agility of the human-computer interaction system based on the response delay of the human-computer interaction system to eye tracking, gesture recognition, voice recognition, electromyography acquisition and environmental perception, combined with the weight coefficient of each response delay indicator;

[0087] The fourth step includes the following steps:

[0088] Step 4.1: Compare each of the eye tracking response delay, gesture recognition response delay, speech recognition response delay, electromyography acquisition response delay and environmental perception response delay to obtain a relative importance index, and use the method in the first step to determine the weight vector W'=(w'1, w'2, w'3, w'4, w'5) of the response delay index;

[0089] Step 4.2: Test the eye tracking response delay, gesture recognition response delay, speech recognition response delay, electromyography acquisition response delay and environmental perception response delay in the human-computer interaction system, and perform normalization to obtain the response delay vector D = (delay1, delay2, delay3, delay4, delay5) T ;

[0090] Step 4.3: According to the response delay weight in step 4.1 and the response delay vector in step 4.2, the agility of the human-computer interaction system is evaluated. The evaluation result calculation formula is: Among them, delay j is the response delay index corresponding to functional dimension j, w' j is the weight coefficient of the response delay index corresponding to functional dimension j. If only eye tracking, gesture recognition, speech recognition, electromyography acquisition and environmental perception functions are considered, then n = 5;

[0091] Step 5: Determine the accuracy of the human-computer interaction system for eye tracking, gesture recognition, speech recognition, electromyography acquisition and environmental perception, and evaluate the accuracy of the human-computer interaction system by combining the weight coefficients of various accuracy indicators;

[0092] The fifth step specifically includes the following steps:

[0093] Step 5.1: Compare the eye tracking accuracy, gesture recognition accuracy, speech recognition accuracy, electromyography acquisition accuracy and environmental perception accuracy to obtain relative importance indicators, and use the method in the first step to determine the weight vector W”=(w”1,w”2,w”3,w”4,w”5) of the accuracy indicator;

[0094] Step 5.2: Test the recognition accuracy of eye tracking, gesture recognition, speech recognition, electromyography and environmental perception respectively, and construct the accuracy vector A = (accu1, accu2, accu3, accu4, accu5) T ;

[0095] Step 5.3: According to the accuracy weight in step 5.1 and the accuracy vector in step 5.2, the accuracy of the human-computer interaction system is evaluated. The evaluation result calculation formula is: Among them, accu j is the accuracy index corresponding to the function index j, w” j is the weight coefficient of the accuracy index corresponding to the functional index j; if only eye tracking, gesture recognition, speech recognition, electromyography acquisition and environmental perception functions are considered, then n = 5;

[0096] Step 6: Based on the evaluation results of the functional diversity, agility and accuracy of the human-computer interaction system, as well as the evaluation weight coefficients corresponding to the functional diversity, agility and accuracy, conduct a comprehensive analysis and evaluation of the effectiveness of the human-computer interaction system;

[0097] The sixth step more specifically includes the following steps:

[0098] Step 6.1: Compare the evaluation results of functional diversity, human-computer interaction agility, and human-computer interaction accuracy to obtain relative importance indicators, and use the method in the first step to determine the evaluation weight vector Weight = (weight1, weight2, weight3);

[0099] Step 6.2: Construct a multi-dimensional evaluation result vector F = (F1, F2, F3) T ;

[0100] Step 6.3: Based on the evaluation weight vector Weight in step 6.1 and the evaluation result vector F in step 6.2, the effectiveness of the human-computer interaction system is comprehensively evaluated. The evaluation result calculation formula is:

[0101] Example 2

[0102] The embodiment of the present application provides a human-computer interaction system effectiveness evaluation system based on multimodal data, including:

[0103] Functional diversity evaluation module, used to analyze the functional support of each functional dimension, and obtain the functional diversity evaluation of the human-computer interaction system by combining the relative weight value of each functional dimension;

[0104] The agility evaluation module is used to calculate the response delay corresponding to each functional dimension of the human-computer interaction system, and obtain the agility evaluation of the human-computer interaction system by combining the weight coefficients of various response delay indicators;

[0105] The accuracy evaluation module is used to calculate the accuracy corresponding to the functional dimension of the human-computer interaction system, and obtain the accuracy evaluation of the human-computer interaction system by combining the weight coefficients of various accuracy indicators;

[0106] The overall evaluation module is used to evaluate the functional diversity, agility and accuracy of the human-computer interaction system. It combines the evaluation weight coefficients corresponding to functional diversity, agility and accuracy to make an overall evaluation of the effectiveness of the human-computer interaction system based on multimodal data.

[0107] Among them, each functional dimension includes at least eye tracking function, gesture recognition function, voice recognition function, electromyography acquisition function and environmental perception function.

[0108] Further preferably, the functional diversity evaluation module includes a first weight acquisition unit, which is used to determine the number n of indicator factors of each functional dimension, compare the importance of every two functional supports in the n indicator factors, and construct a weight matrix corresponding to the functional support; normalize the weight matrix corresponding to the functional support by column, and then add them by row to obtain the added weight matrix corresponding to the functional support; normalize the added matrices corresponding to the functional support to obtain the relative weight values ​​of each functional dimension;

[0109] The agility evaluation module includes a second weight acquisition unit, which is used to determine the number n of indicator factors of each functional dimension, compare the importance of every two response delays in the n indicator factors, and construct a weight matrix corresponding to the response delay; normalize the weight matrix corresponding to the response delay by column, and then add them by row to obtain a weight matrix after the response delays are added; normalize the weight matrix after the response delays are added to obtain the weight coefficient of the response delay indicator;

[0110] The accuracy evaluation module includes a third weight acquisition unit, which is used to determine the number n of indicator factors of each functional dimension, compare the importance of every two accuracies of the n indicator factors, and construct a weight matrix corresponding to the accuracy; normalize the weight matrix corresponding to the accuracy by column, and then add them by row to obtain the weight matrix after the accuracy is added; normalize the weight matrix after the accuracy is added to obtain the weight coefficient of the accuracy indicator.

[0111] Further preferably, the indicator factors of the eye tracking function include the ability to acquire gaze time, the ability to detect line of sight movement distance, the ability to detect pupil changes and the ability to acquire residence time; the indicator factors of the gesture recognition function include the ability to acquire finger movement path, the ability to detect movement speed, the ability to acquire gesture shape and the ability to acquire duration; the indicator factors of the speech recognition function include the ability to acquire timbre, the ability to acquire pitch and the ability to segment speech; the indicator factors of the electromyography acquisition function include the ability to acquire electromyography signals, the ability to acquire nerve conduction velocity and the ability to acquire muscle dynamic characteristics; the indicator factors of the environmental perception function include the ability to perceive temperature, the ability to perceive sound, the ability to perceive light and the ability to recognize objects.

[0112] Further preferably, the method for acquiring the functional support of each functional dimension in the first weight acquisition unit is: determine whether the human-computer interaction system has the indicator factors of each functional dimension, if so, the support coefficient of the corresponding indicator factor is 1, otherwise it is 0; sum the support coefficients corresponding to the indicator factors in each functional dimension to obtain the functional support of each functional dimension.

[0113] Further preferably, the overall evaluation function of the human-computer interaction system effectiveness in the overall evaluation module is:

[0114] Among them, weight1 is the evaluation weight coefficient of functional diversity; weight2 is the evaluation weight coefficient of agility; weight3 is the evaluation weight coefficient of accuracy; F1 is the evaluation result of functional diversity; F2 is the evaluation result of agility; F3 is the evaluation result of accuracy; S j is the functional support corresponding to functional dimension j, w j is the relative weight value corresponding to functional dimension j; delay j is the response delay index corresponding to functional dimension j, w' j is the weight coefficient of the response delay index corresponding to the functional dimension j; accu j is the accuracy index corresponding to functional dimension j, w” j is the weight coefficient of the accuracy index corresponding to the functional dimension j.

[0115] In summary, compared with the prior art, the present application has the following advantages:

[0116] (1) Compared with the traditional single performance evaluation method, this application constructs a human-computer interaction system effectiveness evaluation model from multiple dimensions such as functional diversity, agility and accuracy, which can comprehensively analyze the effectiveness of the human-computer interaction system.

[0117] (2) According to different application scenarios of human-computer interaction systems, the weights of various indicators are taken into account when evaluating the system performance, so as to make a more accurate evaluation of the performance of the human-computer interaction system.

[0118] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0119] Based on the method in the above embodiment, an embodiment of the present application provides an electronic device. The device may include: at least one memory for storing programs and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the above embodiment.

[0120] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0121] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0122] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0123] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0124] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0125] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0126] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for evaluating the effectiveness of a human-computer interaction system based on multimodal data, characterized in that: The following steps are involved: Step S1: Perform function support analysis on each function dimension, and obtain the function diversity evaluation of the human-computer interaction system by combining the relative weight value of each function dimension; At the same time, the response delay corresponding to each functional dimension of the human-computer interaction system is calculated, and the agility evaluation of the human-computer interaction system is obtained by combining the weight coefficients of various response delay indicators; At the same time, the accuracy of each functional dimension of the human-computer interaction system is calculated, and the accuracy evaluation of the human-computer interaction system is obtained by combining the weight coefficients of various accuracy indicators; Step S2: Based on the functional diversity evaluation, agility evaluation and accuracy evaluation of the human-computer interaction system, combined with the evaluation weight coefficients corresponding to the functional diversity, agility and accuracy, an overall evaluation of the effectiveness of the human-computer interaction system based on multimodal data is performed; Among them, each functional dimension includes at least eye tracking function, gesture recognition function, voice recognition function, electromyography acquisition function and environmental perception function.

2. The human-computer interaction system effectiveness evaluation method according to claim 1, characterized in that: The method for obtaining the relative weight value, the weight coefficient of the response delay indicator, and the weight coefficient of the accuracy indicator specifically includes the following steps: Determine the number n of indicator factors of each functional dimension, compare the importance of each two functional support, response delay and accuracy among the n indicator factors, and construct a weight matrix corresponding to the functional support, response delay and accuracy; The weight matrices corresponding to the function support, response delay, and accuracy are normalized by column, and then added by row to obtain the added weight matrices corresponding to the function support, response delay, and accuracy; The added matrices corresponding to function support, response delay and accuracy are normalized respectively to obtain the relative weight value of each function dimension, the weight coefficient of the response delay index, and the weight coefficient of the accuracy index.

3. The human-computer interaction system effectiveness evaluation method according to claim 2, characterized in that: The indicator factors of eye tracking function include the ability to obtain gaze time, the ability to detect moving distance, the ability to detect pupil changes and the ability to obtain residence time; the indicator factors of gesture recognition function include the ability to obtain finger movement path, the ability to detect movement speed, the ability to obtain gesture shape and the ability to obtain duration; the indicator factors of speech recognition function include the ability to obtain timbre, the ability to obtain pitch and the ability to segment speech; the indicator factors of electromyography acquisition function include the ability to obtain electromyography signals, the ability to obtain nerve conduction velocity and the ability to obtain muscle dynamic characteristics; the indicator factors of environmental perception function include the ability to perceive temperature, the ability to perceive sound, the ability to perceive light and the ability to recognize objects.

4. The human-computer interaction system effectiveness evaluation method according to claim 2 or 3, characterized in that: The method for obtaining the functional support of each functional dimension is as follows: Determine whether the human-computer interaction system has the indicator factors of each functional dimension. If so, the support coefficient of the corresponding indicator factor is 1, otherwise it is 0; sum the support coefficients corresponding to the indicator factors in each functional dimension to obtain the functional support of each functional dimension.

5. The human-computer interaction system effectiveness evaluation method according to claim 1, characterized in that: Step S2 specifically includes the following steps: The evaluation results of each of the two items of functional diversity, agility and accuracy are compared to obtain the relative importance index, and the evaluation weight vector Weight=(weight1,weight2,weight3) of functional diversity, agility and accuracy is constructed. The evaluation result vector F=(F1,F2,F3) of the functional diversity evaluation, agility evaluation and accuracy evaluation of the human-computer interaction system is combined. T , to evaluate the effectiveness of the human-computer interaction system Among them, weight1 is the evaluation weight coefficient of functional diversity; weight2 is the evaluation weight coefficient of agility; weight3 is the evaluation weight coefficient of accuracy; F1 is the evaluation result of functional diversity; F2 is the evaluation result of agility; F3 is the evaluation result of accuracy; S j is the functional support corresponding to functional dimension j, w j is the relative weight value corresponding to functional dimension j; delay j is the response delay index corresponding to functional dimension j, w' j is the weight coefficient of the response delay index corresponding to the functional dimension j; accu j is the accuracy index corresponding to functional dimension j, w” j is the weight coefficient of the accuracy index corresponding to functional dimension j; n is the total number of functional dimensions.

6. A human-computer interaction system effectiveness evaluation system based on multimodal data, characterized in that: include: Functional diversity evaluation module, used to analyze the functional support of each functional dimension, and obtain the functional diversity evaluation of the human-computer interaction system by combining the relative weight value of each functional dimension; The agility evaluation module is used to calculate the response delay corresponding to each functional dimension of the human-computer interaction system, and obtain the agility evaluation of the human-computer interaction system by combining the weight coefficients of various response delay indicators; The accuracy evaluation module is used to calculate the accuracy corresponding to the functional dimension of the human-computer interaction system, and obtain the accuracy evaluation of the human-computer interaction system by combining the weight coefficients of various accuracy indicators; The overall evaluation module is used to evaluate the functional diversity, agility and accuracy of the human-computer interaction system. It combines the evaluation weight coefficients corresponding to functional diversity, agility and accuracy to make an overall evaluation of the effectiveness of the human-computer interaction system based on multimodal data. Among them, each functional dimension includes at least eye tracking function, gesture recognition function, voice recognition function, electromyography acquisition function and environmental perception function.

7. The human-computer interaction system effectiveness evaluation system according to claim 6, characterized in that: The functional diversity evaluation module includes a first weight acquisition unit, which is used to determine the number n of indicator factors of each functional dimension, compare the importance of each two functional supports of the n indicator factors, and construct a weight matrix corresponding to the functional support; normalize the weight matrix corresponding to the functional support by column, and then add them by row to obtain the added weight matrix corresponding to the functional support; normalize the added matrices corresponding to the functional support to obtain the relative weight values ​​of each functional dimension; The agility evaluation module includes a second weight acquisition unit, which is used to determine the number n of indicator factors of each functional dimension, compare the importance of every two response delays in the n indicator factors, and construct a weight matrix corresponding to the response delay; The weight matrices corresponding to the response delays are normalized by columns, and then added by rows to obtain the weight matrix after the response delays are added; the weight matrix after the response delays are added is normalized to obtain the weight coefficient of the response delay indicator; The accuracy evaluation module includes a third weight acquisition unit, which is used to determine the number n of indicator factors of each functional dimension, compare the importance of each two accuracy rates of the n indicator factors, and construct a weight matrix corresponding to the accuracy rate; normalize the weight matrix corresponding to the accuracy rate by column, and then add them by row to obtain the weight matrix after the accuracy rate is added; The weight matrix after adding the accuracies is normalized to obtain the weight coefficient of the accuracy index.

8. The human-computer interaction system effectiveness evaluation system according to claim 7, characterized in that: The indicator factors of eye tracking function include the ability to obtain gaze time, the ability to detect line of sight movement distance, the ability to detect pupil changes and the ability to obtain dwell time; the indicator factors of gesture recognition function include the ability to obtain finger movement path, the ability to detect movement speed, the ability to obtain gesture shape and the ability to obtain duration; the indicator factors of speech recognition function include the ability to obtain timbre, the ability to obtain pitch and the ability to segment speech; the indicator factors of electromyography acquisition function include the ability to obtain electromyography signals, the ability to obtain nerve conduction velocity and the ability to obtain muscle dynamic characteristics; the indicator factors of environmental perception function include the ability to perceive temperature, the ability to perceive sound, the ability to perceive light and the ability to recognize objects.

9. The human-computer interaction system effectiveness evaluation system according to claim 8, characterized in that: The method for obtaining the functional support of each functional dimension in the first weight acquisition unit is: determine whether the human-computer interaction system has the indicator factors of each functional dimension. If so, the support coefficient of the corresponding indicator factor is 1, otherwise it is 0; sum the support coefficients corresponding to the indicator factors in each functional dimension to obtain the functional support of each functional dimension.

10. The human-computer interaction system effectiveness evaluation system according to claim 6, characterized in that: The overall evaluation function of the human-computer interaction system effectiveness in the overall evaluation module is: Among them, weight1 is the evaluation weight coefficient of functional diversity; weight2 is the evaluation weight coefficient of agility; weight3 is the evaluation weight coefficient of accuracy; F1 is the evaluation result of functional diversity; F2 is the evaluation result of agility; F3 is the evaluation result of accuracy; S j is the functional support corresponding to functional dimension j, w j is the relative weight value corresponding to functional dimension j; delay j is the response delay index corresponding to functional dimension j, w' j is the weight coefficient of the response delay index corresponding to the functional dimension j; accu j is the accuracy index corresponding to functional dimension j, w” j is the weight coefficient of the accuracy index corresponding to functional dimension j; n is the total number of functional dimensions.

Images