Performance evaluation method and system for multi-modal man-machine interaction

Abstract

The invention belongs to the technical field of man-machine interaction, and provides a multi-modal man-machine interaction performance evaluation method and system, and the method comprises the steps: obtaining real-time multi-modal man-machine interaction data which at least comprises gesture, eye and voice interaction data and a user physiological signal; preprocessing the obtained multi-modal man-machine interaction data to obtain target interaction data; extracting gesture feature data, eye feature data, voice feature data and user physiological feature data in the obtained target interaction data; and according to the extracted gesture feature data, eye feature data, voice feature data and user physiological feature data, and a performance evaluation model based on a long and short term memory network, performing performance evaluation of the multi-modal man-machine interaction to obtain an evaluation result of the performance of the multi-modal man-machine interaction.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-computer interaction, and in particular relates to a performance evaluation method and system for multimodal human-computer interaction. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In a multimodal human-computer interaction system, data is the core, which can not only help the system understand the user's intentions and needs more accurately, but also provide strong support for the optimization and improvement of the system. Therefore, how to effectively collect, process and analyze multimodal data has become the key to the development of multimodal human-computer interaction technology.

[0004] The data used to evaluate the performance of multimodal human-computer interaction is relatively single, which makes it impossible to comprehensively and accurately evaluate the performance of multimodal human-computer interaction. Therefore, in order to evaluate the performance of multimodal human-computer interaction systems, a comprehensive, accurate and efficient evaluation method is needed. However, due to the complexity and diversity of multimodal data, traditional evaluation methods are often unable to cope with it. Therefore, it is urgent to develop an evaluation method that can comprehensively consider multiple modal data, extract key features and output accurate evaluation results. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a performance evaluation method and system for multimodal human-computer interaction. By integrating multiple interaction modes, refined feature extraction and efficient evaluation model construction, a comprehensive and accurate evaluation of the performance of the human-computer interaction system is achieved, providing strong support for the design and optimization of the system.

[0006] According to some embodiments, a first solution of the present invention provides a performance evaluation method for multimodal human-computer interaction, which adopts the following technical solution: A performance evaluation method for multimodal human-computer interaction, comprising: Acquire real-time multimodal human-computer interaction data, wherein the multimodal human-computer interaction data at least includes gesture, eye, voice interaction data and user physiological signals; Preprocessing the acquired multimodal human-computer interaction data to obtain target interaction data; Extracting gesture feature data, eye feature data, voice feature data, and user physiological feature data from the obtained target interaction data; The performance of multimodal human-computer interaction is evaluated based on the extracted gesture feature data, eye feature data, voice feature data and user physiological feature data, and a performance evaluation model based on long short-term memory network to obtain an evaluation result of the performance of multimodal human-computer interaction.

[0007] As a further technical limitation, the process of preprocessing the acquired multimodal human-computer interaction data to obtain target interaction data is: Identify and remove outliers in the interaction data, and use interpolation to fill in missing interaction data based on the distribution characteristics of the interaction data to obtain initial interaction data; The initial interaction data is segmented to analyze data performance in different intervals and obtain target interaction data.

[0008] As a further technical limitation, the process of extracting the gesture feature data, eye feature data, voice feature data and user physiological feature data from the obtained target interaction data is: Filter out the images containing gestures and eyes in the target interaction data, use morphology to process the images containing gestures and eyes, and obtain gesture contours and eye contours through regional connectivity; Perform reverse expansion processing on the gesture contour and the eye contour to obtain gesture feature data and eye feature data; For the voice interaction data, frame processing is performed to determine the spectrum signal corresponding to the voice interaction data, and based on the relationship between the preset spectrum signal and the image pixel value, the spectrum signal is mapped into an image, and the voice feature data is obtained by using an image processing method based on the mapped image; Analyze the period of the physiological signal of the user to determine the time domain information, and use Fourier transform to convert the physiological signal of the user into frequency domain information; Based on the time domain information and frequency domain information, the changes of the user's physiological signals at different times and frequencies are analyzed to obtain the user's physiological characteristic data.

[0009] As a further technical limitation, the gesture feature data includes but is not limited to the position, direction and angle geometric features of the fingers and palms, the speed, acceleration, trajectory dynamic features of the gestures, and the contour, edge, texture shape features of the gestures; The eye feature data includes but is not limited to the coordinate points where the user's eyes are fixed on, the time the user is fixed on the target area or object, and the trajectory and speed of the user's eye scans; The speech feature data includes but is not limited to the Mel frequency cepstral coefficients of the speech signal and the semantic features of the speech content; The user's physiological characteristic data includes, but is not limited to, extracting characteristics reflecting cognitive load from EEG, heart rate, and skin conductance physiological signals.

[0010] As a further technical limitation, the process of constructing the performance evaluation model based on the long short-term memory network includes: Selecting a training set from the target interaction data, and inputting gesture feature data, eye feature data, voice feature data and user's physiological feature data in different time domains in the training set into an input layer of an evaluation model; Input the output of the input layer and the short-term memory network layer in the evaluation model, and input the output of the short-term memory network layer into the long-term memory network layer in the evaluation model to obtain the output of the long-term memory network layer; Inputting the output of the long-term time memory network layer into the prediction layer in the evaluation model, and outputting the performance evaluation result of the human-computer interaction at the last time point in the time domain window corresponding to the training set; The loss function between the performance evaluation result of human-computer interaction and the performance evaluation result of historical human-computer interaction corresponding to the training set is calculated, and the parameters of the evaluation model are adjusted according to the result of the loss function until a preset iteration stop condition is reached to obtain the evaluation model.

[0011] Furthermore, the process of the performance evaluation of the multimodal human-computer interaction is as follows: Inputting the gesture feature data, eye feature data, voice feature data and user's physiological feature data into an input layer of a pre-built evaluation model, and sliding convolution kernels on the gesture feature data, eye feature data, voice feature data and user's physiological feature data to extract local features; The feature map output by the input layer is downsampled using a short-term time memory network layer, and the downsampling result is input into the long-term time memory network layer for global integration and mapping to the sample label space. The prediction layer is used to output the prediction probability corresponding to the gesture feature data, eye feature data, voice feature data and the user's physiological feature data.

[0012] According to some embodiments, a second solution of the present invention provides a performance evaluation system for multimodal human-computer interaction, which adopts the following technical solution: A multimodal human-computer interaction performance evaluation system, comprising: An acquisition module is configured to acquire real-time multimodal human-computer interaction data, wherein the multimodal human-computer interaction data at least includes gesture, eye, voice interaction data and user physiological signals; pre-process the acquired multimodal human-computer interaction data to obtain target interaction data; An extraction module, which is configured to extract gesture feature data, eye feature data, voice feature data and user physiological feature data from the obtained target interaction data; An evaluation module is configured to perform a performance evaluation of multimodal human-computer interaction based on the extracted gesture feature data, eye feature data, voice feature data and user physiological feature data, and a performance evaluation model based on a long short-term memory network, to obtain an evaluation result of the performance of the multimodal human-computer interaction.

[0013] According to some embodiments, a third solution of the present invention provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium stores a program, which, when executed by a processor, implements the steps in a multimodal human-computer interaction performance evaluation method as described in the first solution of the present invention.

[0014] According to some embodiments, a fourth solution of the present invention provides an electronic device, which adopts the following technical solution: An electronic device comprises a memory, a processor and a program stored in the memory and running on the processor, wherein when the processor executes the program, the steps in a performance evaluation method for multimodal human-computer interaction as described in the first solution of the present invention are implemented.

[0015] According to some embodiments, a fifth solution of the present invention provides a computer program product, which adopts the following technical solution: A computer program product includes software codes, wherein the program in the software codes executes the steps in the performance evaluation method of multimodal human-computer interaction as described in the first solution of the present invention.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The present invention comprehensively considers multiple modal data, extracts key features of the considered multimodal data to perform performance evaluation of human-computer interaction, realizes a comprehensive and accurate evaluation of the performance of the human-computer interaction system, and provides strong support for the design and optimization of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings in the specification that constitute a part of this embodiment are used to provide a further understanding of this embodiment. The schematic embodiments of this embodiment and their descriptions are used to explain this embodiment and do not constitute improper limitations on this embodiment.

[0018] Figure 1 is a flow chart of a performance evaluation method for multimodal human-computer interaction in Embodiment 1 of the present invention; Figure 2 This is a structural block diagram of a multimodal human-computer interaction performance evaluation system in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0019] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0020] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0021] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0022] In the present invention, terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings. They are relational words determined only for the convenience of describing the structural relationships of the various parts or elements of the present invention, and do not specifically refer to any part or element in the present invention and should not be understood as limitations on the present invention.

[0023] In the present invention, terms such as "fixed connection", "connected", "connection", etc. should be understood in a broad sense, indicating that it can be fixedly connected, integrally connected or detachably connected; it can be directly connected or indirectly connected through an intermediate medium. For relevant scientific research or technical personnel in this field, the specific meanings of the above terms in the present invention can be determined according to specific circumstances, and they cannot be understood as limitations on the present invention.

[0024] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0025] Embodiment 1 Embodiment 1 of the present invention introduces a performance evaluation method for multimodal human-computer interaction.

[0026] like Figure 1 A performance evaluation method for multimodal human-computer interaction is shown, comprising the following steps: Step 100: collecting interaction data of the user in the process of using the multimodal human-computer interaction system in real time, and processing the interaction data to obtain target interaction data, wherein the interaction data includes but is not limited to gesture, eye, voice interaction data and user's physiological signals; Step 101: extracting gesture feature data, eye feature data, voice feature data and user's physiological feature data from the target interaction data; Step 102: Input the gesture feature data, eye feature data, voice feature data, and user's physiological feature data into a pre-built evaluation model to obtain an evaluation result of the performance of the multimodal human-computer interaction.

[0027] As one or more implementation modes, in step 100, outliers in the interaction data are identified and eliminated, and missing interaction data are filled by interpolation method according to the distribution characteristics of the interaction data to obtain initial interaction data; the initial interaction data is segmented to analyze the data performance in different intervals to obtain target interaction data.

[0028] As one or more implementation modes, in step 101, images containing gestures and eyes in the target interaction data are screened out, and the images containing gestures and eyes are processed using morphology, and gesture contours and eye contours are obtained through regional connectivity; Perform reverse expansion processing on the gesture contour and the eye contour to obtain gesture feature data and eye feature data; For the voice interaction data, frame processing is performed to determine the spectrum signal corresponding to the voice interaction data, and based on the relationship between the preset spectrum signal and the image pixel value, the spectrum signal is mapped into an image, and the voice feature data is obtained by using an image processing method based on the mapped image; Analyze the period of the physiological signal of the user to determine the time domain information, and use Fourier transform to convert the physiological signal of the user into frequency domain information; Based on the time domain information and frequency domain information, the changes of the user's physiological signals at different times and frequencies are analyzed to obtain the user's physiological characteristic data.

[0029] In this embodiment, gesture feature data includes but is not limited to the position, direction and angle geometric features of fingers and palms, the speed, acceleration, trajectory dynamic features of gestures, and the contour, edge, and texture shape features of gestures; eye feature data includes but is not limited to the coordinate points of the user's eye gaze, the user's gaze time on the target area or object, and the trajectory and speed of the user's eye scans; speech feature data includes but is not limited to the Mel-frequency cepstral coefficients of the speech signal and the semantic features of the speech content; the user's physiological feature data includes but is not limited to the extraction of features reflecting cognitive load from EEG, heart rate, and skin conductance physiological signals.

[0030] As one or more implementation modes, in step 102, the evaluation model includes an input layer, a long short-term memory network layer, and a prediction layer, and the construction process of the evaluation model includes: Selecting a training set from the target interaction data, and inputting gesture feature data, eye feature data, voice feature data and user's physiological feature data in different time domains in the training set into an input layer of an evaluation model; Input the output of the input layer and the short-term memory network layer in the evaluation model, and input the output of the short-term memory network layer into the long-term memory network layer in the evaluation model to obtain the output of the long-term memory network layer; Inputting the output of the long-term time memory network layer into the prediction layer in the evaluation model, and outputting the performance evaluation result of the human-computer interaction at the last time point in the time domain window corresponding to the training set; The loss function between the performance evaluation result of human-computer interaction and the performance evaluation result of historical human-computer interaction corresponding to the training set is calculated, and the parameters of the evaluation model are adjusted according to the result of the loss function until a preset iteration stop condition is reached to obtain the evaluation model.

[0031] The loss function in this embodiment is characterized as:

[0032] in, represents the performance evaluation results of human-computer interaction, represents the performance evaluation results of historical human-computer interaction, Represents a pair of training samples.

[0033] The process of performance evaluation of multimodal human-computer interaction in this embodiment is as follows: Inputting the gesture feature data, eye feature data, voice feature data and user's physiological feature data into an input layer of a pre-built evaluation model, and sliding convolution kernels on the gesture feature data, eye feature data, voice feature data and user's physiological feature data to extract local features; The feature map output by the input layer is downsampled using a short-term time memory network layer, and the downsampling result is input into the long-term time memory network layer for global integration and mapping to the sample label space. The prediction layer is used to output the prediction probability corresponding to the gesture feature data, eye feature data, voice feature data and the user's physiological feature data.

[0034] This embodiment comprehensively considers multiple modal data, extracts key features of the considered multimodal data to perform performance evaluation of human-computer interaction, achieves a comprehensive and accurate evaluation of the performance of the human-computer interaction system, and provides strong support for system design and optimization.

[0035] Embodiment 2 Embodiment 2 of the present invention introduces a multi-modal human-computer interaction performance evaluation system.

[0036] like Figure 2 A multimodal human-computer interaction performance evaluation system is shown, comprising: An acquisition module 20 is configured to acquire real-time multimodal human-computer interaction data, wherein the multimodal human-computer interaction data at least includes gesture, eye, voice interaction data and user physiological signals; pre-process the acquired multimodal human-computer interaction data to obtain target interaction data; An extraction module 21, which is configured to extract gesture feature data, eye feature data, voice feature data and user physiological feature data from the obtained target interaction data; The evaluation module 22 is configured to perform a performance evaluation of the multimodal human-computer interaction based on the extracted gesture feature data, eye feature data, voice feature data and user physiological feature data, and a performance evaluation model based on a long short-term memory network, to obtain an evaluation result of the performance of the multimodal human-computer interaction.

[0037] The detailed steps are the same as those of the performance evaluation method for multimodal human-computer interaction provided in Example 1, and will not be repeated here.

[0038] Embodiment 3 Embodiment 3 of the present invention provides a computer-readable storage medium.

[0039] A computer-readable storage medium stores a program, which, when executed by a processor, implements the steps in a performance evaluation method for multimodal human-computer interaction as described in Embodiment 1 of the present invention.

[0040] The detailed steps are the same as those of the performance evaluation method for multimodal human-computer interaction provided in Example 1, and will not be repeated here.

[0041] Embodiment 4 A fourth embodiment of the present invention provides an electronic device.

[0042] An electronic device comprises a memory, a processor and a program stored in the memory and running on the processor, wherein when the processor executes the program, the steps in a performance evaluation method for multimodal human-computer interaction as described in Embodiment 1 of the present invention are implemented.

[0043] The detailed steps are the same as those of the performance evaluation method for multimodal human-computer interaction provided in Example 1, and will not be repeated here.

[0044] Embodiment 5 Embodiment 5 of the present invention provides a computer program product.

[0045] A computer program product includes software codes, wherein the program in the software codes executes the steps in the performance evaluation method for multimodal human-computer interaction as described in the first embodiment of the present invention.

[0046] The detailed steps are the same as those of the performance evaluation method for multimodal human-computer interaction provided in Example 1, and will not be repeated here.

[0047] The above description is only a preferred embodiment of the present embodiment and is not intended to limit the present embodiment. For those skilled in the art, the present embodiment may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present embodiment shall be included in the protection scope of the present embodiment.

Claims

1. A performance evaluation method for multimodal human-computer interaction, characterized in that: include: Acquire real-time multimodal human-computer interaction data, wherein the multimodal human-computer interaction data at least includes gesture, eye, voice interaction data and user physiological signals; Preprocessing the acquired multimodal human-computer interaction data to obtain target interaction data; Extracting gesture feature data, eye feature data, voice feature data, and user physiological feature data from the obtained target interaction data; The performance of multimodal human-computer interaction is evaluated based on the extracted gesture feature data, eye feature data, voice feature data and user physiological feature data, and a performance evaluation model based on long short-term memory network to obtain an evaluation result of the performance of multimodal human-computer interaction.

2. A multimodal human-computer interaction performance evaluation method as claimed in claim 1, characterized in that: The process of preprocessing the acquired multimodal human-computer interaction data to obtain target interaction data is as follows: Identify and remove outliers in the interaction data, and use interpolation to fill in missing interaction data based on the distribution characteristics of the interaction data to obtain initial interaction data; The initial interaction data is segmented to analyze data performance in different intervals and obtain target interaction data.

3. A performance evaluation method for multimodal human-computer interaction as claimed in claim 1, characterized in that: The process of extracting the gesture feature data, eye feature data, voice feature data and user physiological feature data from the obtained target interaction data is as follows: Filter out the images containing gestures and eyes in the target interaction data, use morphology to process the images containing gestures and eyes, and obtain gesture contours and eye contours through regional connectivity; Perform reverse expansion processing on the gesture contour and the eye contour to obtain gesture feature data and eye feature data; For the voice interaction data, frame processing is performed to determine the spectrum signal corresponding to the voice interaction data, and based on the relationship between the preset spectrum signal and the image pixel value, the spectrum signal is mapped into an image, and the voice feature data is obtained by using an image processing method based on the mapped image; Analyze the period of the physiological signal of the user to determine the time domain information, and use Fourier transform to convert the physiological signal of the user into frequency domain information; Based on the time domain information and frequency domain information, the changes of the user's physiological signals at different times and frequencies are analyzed to obtain the user's physiological characteristic data.

4. A performance evaluation method for multimodal human-computer interaction as claimed in claim 1, characterized in that: The gesture feature data includes but is not limited to the position, direction and angle geometric features of the fingers and palms, the speed, acceleration, trajectory dynamic features of the gestures, and the contour, edge, texture shape features of the gestures; The eye feature data includes but is not limited to the coordinate points where the user's eyes are fixed, the time the user is fixed on the target area or object, and the trajectory and speed of the user's eye scans; The speech feature data includes but is not limited to the Mel frequency cepstral coefficients of the speech signal and the semantic features of the speech content; The user's physiological characteristic data includes, but is not limited to, extracting characteristics reflecting cognitive load from EEG, heart rate, and skin conductance physiological signals.

5. A multimodal human-computer interaction performance evaluation method as claimed in claim 1, characterized in that: The process of constructing the performance evaluation model based on the long short-term memory network includes: Selecting a training set from the target interaction data, and inputting gesture feature data, eye feature data, voice feature data and user's physiological feature data in different time domains in the training set into an input layer of an evaluation model; Input the output of the input layer and the short-term memory network layer in the evaluation model, and input the output of the short-term memory network layer into the long-term memory network layer in the evaluation model to obtain the output of the long-term memory network layer; Inputting the output of the long-term time memory network layer into the prediction layer in the evaluation model, and outputting the performance evaluation result of the human-computer interaction at the last time point in the time domain window corresponding to the training set; The loss function between the performance evaluation result of human-computer interaction and the performance evaluation result of historical human-computer interaction corresponding to the training set is calculated, and the parameters of the evaluation model are adjusted according to the result of the loss function until a preset iteration stop condition is reached to obtain the evaluation model.

6. A performance evaluation method for multimodal human-computer interaction as claimed in claim 5, characterized in that: The process of performance evaluation of multimodal human-computer interaction is as follows: Inputting the gesture feature data, eye feature data, voice feature data and user's physiological feature data into an input layer of a pre-built evaluation model, and sliding convolution kernels on the gesture feature data, eye feature data, voice feature data and user's physiological feature data to extract local features; The feature map output by the input layer is downsampled using a short-term time memory network layer, and the downsampling result is input into the long-term time memory network layer for global integration and mapping to the sample label space. The prediction layer is used to output the prediction probability corresponding to the gesture feature data, eye feature data, voice feature data and the user's physiological feature data.

7. A multimodal human-computer interaction performance evaluation system, characterized in that: include: An acquisition module is configured to acquire real-time multimodal human-computer interaction data, wherein the multimodal human-computer interaction data at least includes gesture, eye, voice interaction data and user physiological signals; pre-process the acquired multimodal human-computer interaction data to obtain target interaction data; An extraction module, which is configured to extract gesture feature data, eye feature data, voice feature data and user physiological feature data from the obtained target interaction data; An evaluation module is configured to perform a performance evaluation of multimodal human-computer interaction based on the extracted gesture feature data, eye feature data, voice feature data and user physiological feature data, and a performance evaluation model based on a long short-term memory network, to obtain an evaluation result of the performance of the multimodal human-computer interaction.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of a performance evaluation method for multimodal human-computer interaction as described in any one of claims 1 to 6 are implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the steps of a performance evaluation method for multimodal human-computer interaction as described in any one of claims 1-6 are implemented.

10. A computer program product comprising software code, characterized in that The program in the software code executes the steps of a performance evaluation method for multimodal human-computer interaction as described in any one of claims 1-6.

Images