Man-machine interaction method and system based on multiple modes

Abstract

The invention provides a multi-mode-based man-machine interaction method and system, and relates to the technical field of man-machine interaction. Voice input of a user is received, environment information is collected, facial expressions of the user are captured, and voice features, environment features and facial features are obtained based on the input; utilizing a pre-trained emotion recognition model to generate emotion features of the user, and utilizing a pre-trained user behavior model to generate behavior features according to the current operation information of the user; after the features are obtained, the first weights and the second weights of the features are dynamically adjusted, feature weighted fusion is carried out, and comprehensive features are obtained; the multiple candidate responses are generated based on the comprehensive characteristics, the final response is screened out through a preset scoring mechanism and output, comprehensive understanding of the intention and the emotional state of the user is achieved through comprehensive utilization of multiple data sources, and meanwhile the weight of each characteristic can be dynamically adjusted so that more intelligent and humanized interaction experience can be generated.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and in particular to a multi-modal human-computer interaction method and system thereof. Background Art

[0002] With the rapid development of artificial intelligence technology, human-computer interaction is gradually changing from physical interaction such as keyboard and mouse to more natural and diverse interaction modes. Among them, multimodal human-computer interaction methods have become a hot topic of current research because they can comprehensively utilize multiple data sources such as sound, images, and environmental information to provide users with a richer and more accurate interactive experience.

[0003] Traditional human-computer interaction methods mainly rely on text input or physical button operations. This method is not only cumbersome to operate, but also fails to fully capture the user's emotional state and environmental changes, resulting in a limited interactive experience.

[0004] Therefore, it is necessary to provide a multimodal human-computer interaction method and system to solve the above technical problems. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a multimodal human-computer interaction method and system thereof, which realizes a comprehensive understanding of user intentions and emotional states by comprehensively utilizing multiple data sources, and can dynamically adjust the weights of each feature to generate a more intelligent and humanized interactive experience.

[0006] The present invention provides a multi-modal human-computer interaction method, which comprises the following steps: Receiving a user's voice input, collecting environmental information, and capturing the user's facial expression, and obtaining voice features, environmental features, and facial features based on the voice input, environmental information, and facial expression, respectively; generating an emotion feature of the user based on the voice feature and the facial feature; Based on the user's current operation information, the pre-trained user behavior model is used to generate behavior features; Dynamically adjust the first weight and the second weight of each based on the representation of the speech feature, the environmental feature, the emotional feature and the behavioral feature, and perform weighted fusion of the features based on the first weight and the second weight to obtain a comprehensive feature, wherein the first weight is a feature importance weight, which is used to reflect the relative importance of different features in the interaction process, and the second weight is an attention weight, which is used to adjust the influence of different features in the fusion process; A plurality of candidate responses are generated according to the comprehensive features, and the plurality of candidate responses are screened using a preset scoring mechanism to obtain a final response and output it.

[0007] Preferably, the receiving of the user's voice input, collecting environmental information, and capturing the user's facial expression, and obtaining voice features, environmental features, and facial features based on the voice input, environmental information, and facial expression, respectively, includes: Receive the user's voice input through the voice receiving component, and extract voice features through the voice signal processing method, wherein the voice features include the voice pitch, volume, speaking speed and voice content; Collect environmental information using environmental sensors and extract environmental features through environmental data analysis methods; The user's facial expression is captured by a facial recognition camera component, and the facial image is analyzed using an image processing method to identify and extract facial features, including the open and closed state of the eyes, the shape of the eyebrows, the degree of opening and closing of the mouth, and the overall facial expression type.

[0008] Preferably, generating the user's emotional features based on the voice features and facial features includes: Inputting the extracted speech features and facial features into a pre-trained emotion recognition model, wherein the emotion recognition model is a convolutional neural network using a deep learning architecture to capture emotional clues in speech and facial expressions; The emotion recognition model is used to jointly analyze the input voice features and facial features, and the probability distribution of the user's emotion categories is calculated and output through the trained weight and bias parameters, wherein the user's emotion categories include happiness, sadness, anger, surprise, fear and neutrality; According to the output probability distribution of the user emotion categories, the user emotion category with the highest probability is selected as the user's current emotion, and an emotion feature vector of the user is generated, wherein the emotion feature vector contains the emotion category.

[0009] Preferably, generating behavior features using a pre-trained user behavior model according to the user's current operation information includes: The user operation monitoring component is used to capture and record the user's current operation information in real time, and organize the user's current operation information into a user operation sequence in chronological order, wherein the current operation information includes the user's interaction history with the interactive elements on the interface; Inputting the user operation sequence into a pre-trained user behavior model, wherein the user behavior model is trained based on historical user behavior data and has learned and understood the relationship between user operations and operation intentions; The user operation sequence is encoded and feature extracted using the user behavior model to generate a behavior feature vector, wherein the behavior feature vector contains feature elements for reflecting operation intentions.

[0010] Preferably, the adjustment of the first weight includes: Capturing and parsing the current representation data of the speech features, environmental features, emotional features and behavioral features in real time, wherein the current representation data includes the quantitative value of each feature, the rate of change, and the degree of association with the current interaction context; Based on the current representation data of each feature, the importance score of each feature in the current interaction context is calculated using statistical analysis methods; According to the calculated importance score, weights are assigned to the voice features, environmental features, emotional features and behavioral features to obtain a first weight for each feature, wherein the importance score is positively correlated with the first weight.

[0011] Preferably, the adjustment of the second weight includes: Capturing and recording in real time the reaction time of the user after receiving the voice command issued by the interactive device, wherein the reaction time is used for the user's response speed to the current voice command; Using the attention allocation model and combining the reaction time, the user's attention allocation degree to each feature in the current interaction context is calculated; According to the attention allocation degree, the second weights of the speech features, environmental features, emotional features and behavioral features are dynamically adjusted, wherein the attention score is positively correlated with the second weight.

[0012] Preferably, the acquisition of the comprehensive features includes: According to the first weight and the second weight of each feature obtained by real-time adjustment, a feature weight matrix is ​​constructed and normalized, wherein the rows of the feature weight matrix represent each feature, and the columns represent the first weight or the second weight, wherein the feature weight matrix is: in, and are the first weight and the second weight of the speech feature respectively, and are the first weight and the second weight of the speech feature respectively, and are the first weight and the second weight of the speech feature respectively, and are the first weight and the second weight of the speech feature respectively, is the feature weight matrix; Based on the normalized feature weight matrix, each feature is weighted; The weighted features are fused through feature concatenation method to generate a comprehensive feature vector.

[0013] Preferably, the generating of multiple candidate responses according to the comprehensive features, and screening the multiple candidate responses using a preset scoring mechanism to obtain and output a final response includes: Based on the comprehensive feature vector, generate a plurality of candidate responses using a pre-trained response generation model; Using the pre-trained sentiment analysis model, calculate the response sentiment feature vector of each candidate response; Calculate the emotional similarity between the response emotional feature vector of each candidate response and the user's current emotional feature vector, wherein the emotional similarity is expressed in cosine degree; Generate a sentiment matching score for each candidate response based on the calculated sentiment similarity; According to the sentiment matching score, the candidate response with the highest score is selected as the final response and output.

[0014] The present invention also provides a multi-modal human-computer interaction system for executing a multi-modal human-computer interaction method, the interaction system comprising: A feature acquisition module, used to receive a user's voice input, collect environmental information, and capture the user's facial expression, and acquire voice features, environmental features, and facial features based on the voice input, environmental information, and facial expression, respectively; An emotion feature generation module, used to generate the user's emotion features based on the voice features and facial features; A behavior feature generation module is used to generate behavior features based on the user's current operation information using a pre-trained user behavior model; A comprehensive feature acquisition module, used to dynamically adjust the first weight and the second weight of each of the voice feature, the environmental feature, the emotional feature and the behavioral feature based on the representation, and perform feature weighted fusion based on the first weight and the second weight to obtain a comprehensive feature, wherein the first weight is a feature importance weight, which is used to reflect the relative importance of different features in the interaction process, and the second weight is an attention weight, which is used to adjust the influence of different features in the fusion process; The response generation module is used to generate multiple candidate responses according to the comprehensive features, and screen the multiple candidate responses using a preset scoring mechanism to obtain and output a final response.

[0015] Compared with the related art, the multimodal human-computer interaction method and system provided by the present invention have the following beneficial effects: The present invention receives the user's voice input, collects environmental information, and captures the user's facial expression, and obtains voice features, environmental features, and facial features based on these inputs; then, the user's emotional features are generated using a pre-trained emotion recognition model, and behavioral features are generated using a pre-trained user behavior model based on the user's current operation information; after obtaining these features, the present invention dynamically adjusts the respective first weights and second weights, and performs weighted feature fusion to obtain comprehensive features; finally, multiple candidate responses are generated based on the comprehensive features, and a preset scoring mechanism is used to screen out and output the final response. The present invention achieves a comprehensive understanding of the user's intentions and emotional states by comprehensively utilizing multiple data sources, and can dynamically adjust the weights of each feature to generate a more intelligent and humanized interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of a multi-modal human-computer interaction method provided by the present invention; Figure 2 A module diagram of a multi-modal human-computer interaction system provided by the present invention. DETAILED DESCRIPTION

[0017] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0018] It should also be noted that, for ease of description, only the parts related to the present invention, but not all of the contents, are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the operations (or steps) as sequential processes, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0019] Embodiment 1 The present invention provides a multi-modal human-computer interaction method, referring to Figure 1 As shown, the interaction method comprises the following steps: S1: receiving a user's voice input, collecting environmental information, and capturing the user's facial expression, and obtaining voice features, environmental features, and facial features based on the voice input, environmental information, and facial expression, respectively.

[0020] Specifically, step S1 includes the following steps: S11: receiving a user's voice input through a voice receiving component, and extracting voice features through a voice signal processing method, wherein the voice features include the voice's pitch, volume, speaking speed, and voice content.

[0021] In this embodiment, the voice receiving component may be, but is not limited to, a microphone array, which is used to capture the user's voice input. The voice signal processing method mainly includes the following steps: First, the captured speech signal is preprocessed, including denoising, noise reduction and normalization, to remove environmental noise and improve signal quality.

[0022] Multiple features are extracted from the preprocessed speech signal, including pitch, volume, speech rate, and speech content. Pitch is extracted through spectrum analysis, volume is extracted through energy analysis, speech rate is extracted through duration analysis, and speech content is converted into text through automatic speech recognition (ASR) technology.

[0023] The various extracted features are combined into a feature vector for subsequent processing and analysis.

[0024] Through these steps, the user's voice features can be accurately captured and analyzed, providing reliable data support for subsequent emotion recognition and interaction. This method not only improves the accuracy of speech recognition, but also can capture the user's emotional changes, thereby improving the intelligence level of human-computer interaction.

[0025] S12: Collect environmental information using environmental sensors and extract environmental features through environmental data analysis methods.

[0026] In this embodiment, the environmental sensors include temperature sensors, humidity sensors, light sensors, and sound sensors, which are used to collect environmental information around the user. The environmental data analysis method mainly includes the following steps: Environmental data, including temperature, humidity, light intensity, and background noise, are collected in real time through various environmental sensors.

[0027] The collected environmental data are preprocessed, including filtering, smoothing and standardization, to remove noise and outliers and improve data quality.

[0028] Extract multiple features from the preprocessed environmental data, including ambient temperature, humidity, light intensity, and background noise level. These features can reflect the user's current environmental conditions.

[0029] The various environmental features extracted are combined into a feature vector through feature splicing method for subsequent processing and analysis.

[0030] Through these steps, we can fully understand the environmental conditions of the user, providing important background information for subsequent behavior analysis and emotion recognition. This method can not only improve the system's environmental perception ability, but also dynamically adjust the interaction strategy according to environmental changes, thus improving the user experience.

[0031] S13: Capturing the user's facial expression through a facial recognition camera component, and analyzing the facial image using an image processing method to identify and extract facial features, wherein the facial features include the opening and closing state of the eyes, the shape of the eyebrows, the opening and closing degree of the mouth, and the overall facial expression type.

[0032] In this embodiment, the facial recognition camera assembly may be, but is not limited to, a high-resolution camera for capturing a user's facial image. The image processing method mainly includes the following steps: The user's facial image is captured in real time through the camera to ensure that the image is clear and unobstructed.

[0033] The collected facial images are preprocessed, including grayscale, denoising and standardization, to improve image quality and reduce noise interference.

[0034] Use face detection algorithms (Haar cascade classifiers or deep learning models) to detect and locate facial key points, including eyes, eyebrows, mouth, etc.

[0035] Extract multiple facial features from the detected key points, including the open and closed state of the eyes, the shape of the eyebrows, the degree of mouth opening and closing, and the overall facial expression type. These features can be extracted through the analysis of geometric features and texture features.

[0036] The various facial features extracted are also combined into a feature vector through the feature splicing method for subsequent processing and analysis.

[0037] S2: Generate the user's emotional features based on the voice features and facial features.

[0038] Specifically, step S2 includes the following steps: S21: Inputting the extracted speech features and facial features into a pre-trained emotion recognition model, wherein the emotion recognition model is a convolutional neural network using a deep learning architecture, and is used to capture emotional clues in speech and facial expressions.

[0039] In this embodiment, the emotion recognition model is a convolutional neural network (CNN) based on deep learning, which is used to capture emotional clues in speech and facial expressions. The specific working principle and process are as follows: The structure of the emotion recognition model includes: Input layer: Receives feature vectors passed in from the speech and facial feature extraction modules.

[0040] Convolution layer: Perform convolution operations on the input features through multiple convolution kernels to extract local features.

[0041] Pooling layer: Reduce feature dimensions and retain main features through maximum pooling or average pooling operations.

[0042] Fully connected layer: maps the convolution and pooling features into a fixed-length vector for classification tasks.

[0043] Output layer: Outputs the probability distribution of user emotion categories.

[0044] The features are input into the emotion recognition model, where voice features include pitch, volume, speaking speed and voice content; facial features include the openness or closedness of the eyes, the shape of the eyebrows, the degree of opening or closing of the mouth and the overall facial expression type.

[0045] At the same time, the feature vectors of speech features and facial features are concatenated into a feature vector and input into the emotion recognition model. The model extracts emotional clues from speech and facial features through multi-layer convolution and pooling operations.

[0046] S22: Utilize the emotion recognition model to jointly analyze the input voice features and facial features, calculate and output the probability distribution of the user emotion categories through the trained weights and bias parameters, wherein the user emotion categories include happiness, sadness, anger, surprise, fear and neutrality.

[0047] In this embodiment, the emotion recognition model uses the trained weights and bias parameters to jointly analyze the input voice features and facial features, calculates and outputs the probability distribution of the user's emotion category. The specific working principle and process are as follows: First, feature processing is performed, specifically: the feature vectors of speech features and facial features are concatenated into one feature vector, and the feature vector is normalized to ensure that the feature values ​​are within the same range.

[0048] Then, model inference is performed, specifically as follows: the normalized feature vector is input into the emotion recognition model, high-level features are extracted through multi-layer convolution and pooling operations, the ReLU activation function is used to increase the nonlinear ability of the model, the extracted high-level features are mapped to a vector of fixed length for classification tasks, and the output of the fully connected layer is converted into a probability distribution through the softmax function to represent the probability of the user's emotion category.

[0049] Finally, the probability distribution of the user's emotion categories is output, including the probabilities of happiness, sadness, anger, surprise, fear, and neutral.

[0050] S23: According to the output probability distribution of the user emotion categories, select the user emotion category with the highest probability as the user's current emotion, and generate an emotion feature vector of the user, wherein the emotion feature vector includes the emotion category.

[0051] In this embodiment, according to the probability distribution of the user emotion categories output by the emotion recognition model, the user emotion category with the highest probability is selected as the user's current emotion, and the user's emotion feature vector is generated. The specific working principle and process are as follows: First, a probability distribution analysis is performed, specifically: The probability distribution of the user's emotion categories is obtained from the output of the emotion recognition model, including the probability values ​​of happiness, sadness, anger, surprise, fear, and neutrality, and the user emotion category with the highest probability is selected as the user's current emotion.

[0052] Then, the emotional feature vector is generated, specifically: The selected user emotion category is used as the emotion label, and the emotion feature vector containing the emotion label is generated.

[0053] S3: Generate behavior features using the pre-trained user behavior model based on the user's current operation information.

[0054] Specifically, step S3 includes the following steps: S31: Capture and record the user's current operation information in real time through the user operation monitoring component, and organize the user's current operation information into a user operation sequence in chronological order, wherein the current operation information includes the user's interaction history with the interactive elements on the interface.

[0055] In this embodiment, the user operation monitoring component is used to capture and record the user's current operation information in real time. The specific working principle and process are as follows: First, the user operation monitoring component can be but not limited to input devices such as touch screens, mice, keyboards, etc., which are used to capture the user's operation behavior and capture the user's operation information in real time through these input devices, including clicks, slides, and input text.

[0056] Then, the captured operation information is recorded, including the timestamp of the operation, the operation type (click, slide), and the operation location, and the user's operation information is organized into a user operation sequence in chronological order. For example, the user's operation sequence on an interface may include: click button A → slide the screen → enter text → click button B.

[0057] Finally, the user operation sequence contains the interaction history between the user and the interactive elements on the interface. This information can reflect the user's operating habits and intentions.

[0058] S32: Inputting the user operation sequence into a pre-trained user behavior model, wherein the user behavior model is trained based on historical user behavior data and has learned and understood the relationship between user operations and operation intentions.

[0059] In this embodiment, the user behavior model is a deep learning model trained based on historical user behavior data, which is used to understand and predict user operation intentions. The specific working principle and process are as follows: The structure of the user behavior model includes: Input layer: receives user operation sequences as input.

[0060] Embedding layer: Converts information such as operation type and timestamp into a fixed-length embedding vector.

[0061] Recurrent Neural Network (RNN): RNN is used to model the user operation sequence and capture the temporal dependencies between operations.

[0062] Fully connected layer: maps the output of the RNN to a fixed-length vector for classification tasks.

[0063] Output layer: Outputs the probability distribution or feature vector of the user's operation intention.

[0064] When building a user behavior model, a large amount of historical user behavior data is collected, including the user's operation sequence on the interface and its corresponding operation intentions. The historical data is then cleaned and preprocessed, including removing outliers and filling missing values. The user behavior model is trained using historical user behavior data, and the model parameters are optimized through the back propagation algorithm, so that the model can accurately understand and predict user operation intentions.

[0065] The cross entropy loss function is used to measure the difference between the model's predicted value and the true value.

[0066] When processing the user behavior model, the user operation sequence is input into the pre-trained user behavior model, and the user behavior model outputs the probability distribution or feature vector of the user's operation intention. This information can be used for subsequent behavior analysis and emotion recognition.

[0067] S33: Encoding and feature extracting the user operation sequence using the user behavior model to generate a behavior feature vector, wherein the behavior feature vector includes feature elements for reflecting operation intentions.

[0068] In this embodiment, the user behavior model encodes and extracts features of the user operation sequence to generate a behavior feature vector. The specific working principle and process are as follows: Each operation type and timestamp in the user operation sequence is converted into a fixed-length embedding vector.

[0069] The user operation sequence is temporally modeled through RNN to capture the temporal dependencies between operations.

[0070] The output of the RNN is mapped to a fixed-length feature vector that contains feature elements used to reflect the operation intention.

[0071] The generated behavioral feature vector contains important features of the user operation sequence, which can reflect the user's operation intentions and habits. Each element in the feature vector represents a specific feature, such as operation frequency, operation type distribution, operation time interval, etc.

[0072] S4: Based on the representation of the speech features, environmental features, emotional features and behavioral features, dynamically adjust the respective first weights and second weights, and perform weighted fusion of features based on the first weights and the second weights to obtain comprehensive features, wherein the first weight is the feature importance weight, which is used to reflect the relative importance of different features in the interaction process, and the second weight is the attention weight, which is used to adjust the influence of different features in the fusion process.

[0073] In step S4, the adjustment of the first weight includes: The current characterization data of the speech features, environmental features, emotional features and behavioral features are captured and analyzed in real time, wherein the current characterization data includes the quantitative value of each feature, the rate of change, and the degree of association with the current interaction context.

[0074] In this embodiment, real-time capture and analysis of the user's current feature representation data is the basis for dynamically adjusting feature weights. The specific working principle and process are as follows: Through the voice receiving component, environmental sensor, facial recognition camera component and user operation monitoring component, the user's voice features, environmental features, facial features and behavioral features are captured in real time, the captured data is analyzed, and the quantitative value of each feature is extracted. For example, the pitch, volume, speaking speed and voice content of voice features; the temperature, humidity, light intensity and background noise of environmental features; the eye opening and closing state, eyebrow shape, mouth opening and closing degree and overall expression type of facial features; the operation type, operation location and timestamp of behavioral features.

[0075] For the rate of change, the calculation method is: Set a time window and calculate the change rate of each feature within the time window. The change rate reflects the change trend of the feature over time.

[0076] The rate of change of each feature is calculated by the differential calculation method, where the differential formula is: in, is the rate of change, For the current time point The characteristic value of For the previous time point The characteristic value of is the time interval; For example, the pitch change rate of the speech feature , time point is , time interval is 1 second, and the calculation formula is: in, For the current time point The tone of For the previous time point The tone of Temperature change rate for environmental characteristics , time point is , time interval is 1 second, and the calculation formula is: in, For the current time point The temperature, For the previous time point Temperature; Eye opening and closing change rate for facial features , time point is , time interval is 1 second, and the calculation formula is: in, For the current time point The eye opening and closing value, For the previous time point The eye opening and closing value.

[0077] The degree of context association can be obtained through correlation analysis, which can intuitively reflect the linear relationship between features, specifically: The quantified values ​​and change rates of each captured feature are organized into a feature vector, and the information of the current interaction context is defined, such as the user's current operation intention and emotional state.

[0078] Calculate the Pearson correlation coefficient between each feature and the current interaction context, where the Pearson correlation coefficient The calculation formula is: in, is the eigenvalue of each feature, is the context information value, and are the means of the feature value and context information value, respectively.

[0079] The calculated Pearson correlation coefficient is converted into a correlation score Q, which can be converted into a correlation score Q using the following formula: The correlation score ranges from 0 to 1, and the larger the value, the higher the correlation.

[0080] Based on the current representation data of each feature, the importance score of each feature in the current interaction context is calculated using statistical analysis methods.

[0081] In this embodiment, based on the captured current feature representation data, a statistical analysis method is used to calculate the importance score of each feature. The specific working principle and process are as follows: First, the quantized value, change rate and correlation of each feature captured are organized into a feature vector, and the feature vector is normalized to ensure that each feature value is in the same range to avoid the influence of some feature values ​​that are too large or too small on the results.

[0082] Then, the principal components of each feature are extracted through the principal component analysis method. Principal component analysis can convert high-dimensional feature space into low-dimensional feature space while retaining most of the information of the original data. The contribution of each feature in the principal component is then calculated. The contribution reflects the importance of the feature in the current interaction situation. The contribution can be evaluated by the variance explanation ratio of the principal component.

[0083] The linear regression model is used to evaluate the influence of each feature on the current interaction context. The linear regression model can be used to predict the user's emotional state. At the same time, by training the linear regression model, the absolute value of the coefficient of each feature in the model is calculated, where the absolute value of the coefficient reflects the influence of the feature on the target variable.

[0084] Finally, the importance score of each feature is weighted and calculated by combining the correlation score of the correlation analysis, the contribution of the principal component analysis and the absolute value of the coefficient of the regression analysis. A weight is assigned to each analysis result, and these weights can be adjusted according to actual needs.

[0085] The importance score reflects the relative importance of each feature in the current interaction context. Based on the importance score, a preliminary weight allocation is made to each feature. Features with high importance scores are given higher weights, and vice versa.

[0086] According to the calculated importance score, weights are assigned to the voice features, environmental features, emotional features and behavioral features to obtain a first weight for each feature, wherein the importance score is positively correlated with the first weight.

[0087] In this embodiment, weights are assigned to each feature according to the calculated importance score to generate a first weight. The specific working principle and process are as follows: The calculated importance scores are sorted in order from high to low, and the importance scores are normalized to ensure that the sum of all scores is 1.

[0088] The normalized importance scores are linearly mapped to a weight range (e.g., 0 to 1). Features with high importance scores are given higher first weights, and vice versa.

[0089] At the same time, the first weight of the feature is dynamically adjusted according to the user's real-time feature characterization data. When the user's feature characterization data changes, the importance score is recalculated and the first weight is adjusted.

[0090] In addition, in order to avoid drastic fluctuations in the first weight, the first weight may be smoothed, for example, using an exponentially weighted moving average (EWMA) method.

[0091] In step S4, the adjustment of the second weight includes: The user's reaction time after receiving a voice command from an interactive device is captured and recorded in real time, wherein the reaction time is used to measure the user's response speed to the current voice command.

[0092] In this embodiment, real-time capture and recording of the user's reaction time after receiving the voice command issued by the interactive device is the basis for dynamically adjusting the second weight. The specific working principle and process are as follows: Send voice commands to users through interactive devices, and record the time interval from when the user receives the voice command to when the user responds. The response can be a voice reply, touch screen operation, or any other form of interactive behavior.

[0093] Record the timestamp of the voice command and the timestamp of the user's response, and calculate the user reaction time, which is the time interval from the voice command to the user's response. The reaction time reflects the user's response speed to the current voice command.

[0094] The attention allocation model is used in combination with the reaction time to calculate the user's attention allocation degree to each feature in the current interaction context.

[0095] In this embodiment, the attention allocation model is combined with the user's reaction time to calculate the user's attention allocation degree to each feature in the current interaction context. The specific working principle and process are as follows: The attention allocation model is a deep learning-based model that can dynamically adjust the degree of attention paid to different features based on the user's reaction time.

[0096] The input layer of the attention allocation model receives the user's reaction time data and other feature data (such as voice features, environmental features, emotional features, and behavioral features).

[0097] The attention layer of the attention allocation model calculates the attention weight of each feature through the attention mechanism. The attention weight reflects the user's attention to each feature in the current interaction context.

[0098] The user's reaction time and other feature data are organized into feature vectors, and the feature vectors are normalized to ensure that each feature value is within the same range to avoid the influence of some feature values ​​that are too large or too small on the results.

[0099] The attention weight of each feature is calculated through the attention mechanism. The attention weight reflects the user's attention to each feature.

[0100] According to the attention allocation degree, the second weights of the speech features, environmental features, emotional features and behavioral features are dynamically adjusted, wherein the attention score is positively correlated with the second weight.

[0101] In this embodiment, the second weight of each feature is dynamically adjusted according to the calculated attention allocation degree. The specific working principle and process are as follows: The calculated attention scores are sorted in order from high to low, and the attention scores are normalized to ensure that the sum of all scores is 1.

[0102] The normalized attention scores are linearly mapped to a weight range (e.g., 0 to 1). Features with high attention scores are assigned higher second weights, and vice versa.

[0103] According to the user's real-time reaction time data, the second weight of the feature is dynamically adjusted. When the user's reaction time data changes, the attention score is recalculated and the second weight is adjusted.

[0104] In addition, in order to avoid drastic fluctuations in the second weight, the second weight may be smoothed, for example, using an exponentially weighted moving average (EWMA) method.

[0105] In step S4, the acquisition of the comprehensive features includes: According to the first weight and the second weight of each feature obtained by real-time adjustment, a feature weight matrix is ​​constructed and normalized, wherein the rows of the feature weight matrix represent each feature, and the columns represent the first weight or the second weight, wherein the feature weight matrix is: in, and are the first weight and the second weight of the speech feature respectively, and are the first weight and the second weight of the speech feature respectively, and are the first weight and the second weight of the speech feature respectively, and are the first weight and the second weight of the speech feature respectively, is the feature weight matrix.

[0106] Based on the normalized feature weight matrix, each feature is weighted.

[0107] The weighted features are fused through feature splicing method to generate a comprehensive feature vector, where the splicing method is simple vector splicing.

[0108] S5: Generate multiple candidate responses according to the comprehensive features, and use a preset scoring mechanism to screen the multiple candidate responses to obtain a final response and output it.

[0109] Specifically, step S5 includes the following steps: S51: Based on the comprehensive feature vector, a plurality of candidate responses are generated using a pre-trained response generation model.

[0110] In this embodiment, the response generation model generally includes an input layer, an embedding layer, an encoding layer, a decoding layer, and an output layer. The input layer receives a comprehensive feature vector, the embedding layer converts the feature into an embedding vector, the encoding layer extracts high-level features through an encoder, the decoding layer generates multiple candidate responses, and the output layer outputs the generated response.

[0111] Collect a large amount of historical conversation data, including the user's comprehensive feature vector and the corresponding system response, and clean and preprocess it. Use the historical conversation data to train the response generation model, and optimize the model parameters through the back propagation algorithm so that the model can generate high-quality responses. The loss function usually uses the cross entropy loss function, inputs the comprehensive feature vector into the pre-trained response generation model, and the model outputs multiple candidate responses, which can be text or other forms of feedback.

[0112] S52: Using the pre-trained sentiment analysis model, calculate the response sentiment feature vector of each candidate response.

[0113] In this embodiment, the sentiment analysis model is a deep learning-based model used to identify sentiment features in text.

[0114] Collect historical text data with sentiment labels and perform cleaning and preprocessing.

[0115] The sentiment analysis model is trained using historical text data with sentiment labels, and the model parameters are optimized through the back propagation algorithm so that the model can accurately identify the sentiment features in the text. The loss function usually uses the cross entropy loss function.

[0116] Each generated candidate response text is input into the pre-trained sentiment analysis model, and the model outputs the sentiment feature vector of each candidate response, which contains the probability distribution of the sentiment category.

[0117] S53: Calculate the emotional similarity between the response emotional feature vector of each candidate response and the user's current emotional feature vector, wherein the emotional similarity is represented by cosine degree.

[0118] In this embodiment, the user emotion feature vector is generated from step S23, and the candidate response emotion feature vector is generated from step S52. The cosine similarity is used to calculate the similarity between the two emotion feature vectors, and the emotion similarity score of each candidate response is calculated. The score ranges from 0 to 1, and the larger the value, the higher the emotion similarity.

[0119] S54: Generate a sentiment matching score for each candidate response based on the calculated sentiment similarity.

[0120] In this embodiment, the emotion similarity score of each candidate response calculated in step S53 is normalized to ensure that the sum of all scores is 1. The normalized emotion similarity score is used as the emotion matching score, which reflects the matching degree between the candidate response and the user's current emotion state.

[0121] S55: According to the sentiment matching score, the candidate response with the highest score is selected as the final response and output.

[0122] In this embodiment, the emotion matching scores of all candidate responses are sorted from high to low, the candidate response with the highest emotion matching score is selected as the final response, the selected final response is generated as the system output, and the final response is fed back to the user through voice, text or other forms.

[0123] Embodiment 2 The present invention also provides a multi-modal human-computer interaction system for executing a multi-modal human-computer interaction method, referring to Figure 2 As shown, the interactive system includes: The feature acquisition module 100 is used to receive the user's voice input, collect environmental information and capture the user's facial expression, and obtain voice features, environmental features and facial features based on the voice input, environmental information and facial expression respectively.

[0124] The emotion feature generation module 200 is used to generate the user's emotion features based on the voice features and facial features.

[0125] The behavior feature generation module 300 is used to generate behavior features based on the user's current operation information using a pre-trained user behavior model.

[0126] The comprehensive feature acquisition module 400 is used to dynamically adjust the respective first weights and second weights based on the representation of the speech features, environmental features, emotional features and behavioral features, and perform feature weighted fusion based on the first weights and the second weights to obtain a comprehensive feature, wherein the first weight is the feature importance weight, which is used to reflect the relative importance of different features in the interaction process, and the second weight is the attention weight, which is used to adjust the influence of different features in the fusion process.

[0127] The response generation module 500 is used to generate multiple candidate responses according to the comprehensive features, and screen the multiple candidate responses using a preset scoring mechanism to obtain and output a final response.

[0128] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0129] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, the storage medium including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically-erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0130] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. A multimodal human-computer interaction method, characterized in that: The interactive method comprises the following steps: Receiving a user's voice input, collecting environmental information, and capturing the user's facial expression, and obtaining voice features, environmental features, and facial features based on the voice input, environmental information, and facial expression, respectively; generating an emotion feature of the user based on the voice feature and the facial feature; Based on the user's current operation information, the pre-trained user behavior model is used to generate behavior features; Dynamically adjust the first weight and the second weight of each based on the representation of the speech feature, the environmental feature, the emotional feature and the behavioral feature, and perform weighted fusion of the features based on the first weight and the second weight to obtain a comprehensive feature, wherein the first weight is a feature importance weight, which is used to reflect the relative importance of different features in the interaction process, and the second weight is an attention weight, which is used to adjust the influence of different features in the fusion process; A plurality of candidate responses are generated according to the comprehensive features, and the plurality of candidate responses are screened using a preset scoring mechanism to obtain a final response and output it.

2. The multimodal human-computer interaction method according to claim 1, characterized in that: The receiving of the user's voice input, collecting environmental information, and capturing the user's facial expression, and obtaining voice features, environmental features, and facial features based on the voice input, environmental information, and facial expression, respectively, includes: Receive the user's voice input through the voice receiving component, and extract voice features through the voice signal processing method, wherein the voice features include the voice pitch, volume, speaking speed and voice content; Collect environmental information using environmental sensors and extract environmental features through environmental data analysis methods; The user's facial expression is captured by a facial recognition camera component, and the facial image is analyzed using an image processing method to identify and extract facial features, including the open and closed state of the eyes, the shape of the eyebrows, the degree of opening and closing of the mouth, and the overall facial expression type.

3. The multimodal human-computer interaction method according to claim 2, characterized in that: The generating the user's emotion feature based on the voice feature and the facial feature comprises: Inputting the extracted speech features and facial features into a pre-trained emotion recognition model, wherein the emotion recognition model is a convolutional neural network using a deep learning architecture to capture emotional clues in speech and facial expressions; The emotion recognition model is used to jointly analyze the input voice features and facial features, and the probability distribution of the user's emotion categories is calculated and output through the trained weight and bias parameters, wherein the user's emotion categories include happiness, sadness, anger, surprise, fear and neutrality; According to the output probability distribution of the user emotion categories, the user emotion category with the highest probability is selected as the user's current emotion, and an emotion feature vector of the user is generated, wherein the emotion feature vector contains the emotion category.

4. The multimodal human-computer interaction method according to claim 3, characterized in that: The generating behavior features by using a pre-trained user behavior model according to the user's current operation information includes: The user operation monitoring component is used to capture and record the user's current operation information in real time, and organize the user's current operation information into a user operation sequence in chronological order, wherein the current operation information includes the user's interaction history with the interactive elements on the interface; Inputting the user operation sequence into a pre-trained user behavior model, wherein the user behavior model is trained based on historical user behavior data and has learned and understood the relationship between user operations and operation intentions; The user operation sequence is encoded and feature extracted using the user behavior model to generate a behavior feature vector, wherein the behavior feature vector contains feature elements for reflecting operation intentions.

5. The multimodal human-computer interaction method according to claim 4, characterized in that: The adjustment of the first weight includes: Capturing and parsing the current representation data of the speech features, environmental features, emotional features and behavioral features in real time, wherein the current representation data includes the quantitative value of each feature, the rate of change, and the degree of association with the current interaction context; Based on the current representation data of each feature, the importance score of each feature in the current interaction context is calculated using statistical analysis methods; According to the calculated importance score, weights are assigned to the voice features, environmental features, emotional features and behavioral features to obtain a first weight for each feature, wherein the importance score is positively correlated with the first weight.

6. The multimodal human-computer interaction method according to claim 5, characterized in that: The adjustment of the second weight includes: Capturing and recording in real time the reaction time of the user after receiving the voice command issued by the interactive device, wherein the reaction time is used for the user's response speed to the current voice command; Using the attention allocation model and combining the reaction time, the user's attention allocation degree to each feature in the current interaction context is calculated; According to the attention allocation degree, the second weights of the speech features, environmental features, emotional features and behavioral features are dynamically adjusted, wherein the attention score is positively correlated with the second weight.

7. The multimodal human-computer interaction method according to claim 6, characterized in that: The acquisition of the comprehensive features includes: According to the first weight and the second weight of each feature obtained by real-time adjustment, a feature weight matrix is ​​constructed and normalized, wherein the rows of the feature weight matrix represent each feature, and the columns represent the first weight or the second weight, wherein the feature weight matrix is: in, and are the first weight and the second weight of the speech feature respectively, and are the first weight and the second weight of the speech feature respectively, and are the first weight and the second weight of the speech feature respectively, and are the first weight and the second weight of the speech feature respectively, is the feature weight matrix; Based on the normalized feature weight matrix, each feature is weighted; The weighted features are fused through feature concatenation method to generate a comprehensive feature vector.

8. The multimodal human-computer interaction method according to claim 7, characterized in that: The generating of multiple candidate responses according to the comprehensive features, and screening the multiple candidate responses using a preset scoring mechanism to obtain and output a final response includes: Based on the comprehensive feature vector, generate a plurality of candidate responses using a pre-trained response generation model; Using the pre-trained sentiment analysis model, calculate the response sentiment feature vector of each candidate response; Calculate the emotional similarity between the response emotional feature vector of each candidate response and the user's current emotional feature vector, wherein the emotional similarity is expressed in cosine degree; Generate a sentiment matching score for each candidate response based on the calculated sentiment similarity; According to the sentiment matching score, the candidate response with the highest score is selected as the final response and output.

9. A multimodal human-computer interaction system, used to execute a multimodal human-computer interaction method according to any one of claims 1 to 8, characterized in that: The interactive system comprises: A feature acquisition module, used to receive a user's voice input, collect environmental information, and capture the user's facial expression, and acquire voice features, environmental features, and facial features based on the voice input, environmental information, and facial expression, respectively; An emotion feature generation module, used to generate the user's emotion features based on the voice features and facial features; A behavior feature generation module is used to generate behavior features based on the user's current operation information using a pre-trained user behavior model; A comprehensive feature acquisition module, used to dynamically adjust the first weight and the second weight of each of the voice feature, the environmental feature, the emotional feature and the behavioral feature based on the representation, and perform feature weighted fusion based on the first weight and the second weight to obtain a comprehensive feature, wherein the first weight is a feature importance weight, which is used to reflect the relative importance of different features in the interaction process, and the second weight is an attention weight, which is used to adjust the influence of different features in the fusion process; The response generation module is used to generate multiple candidate responses according to the comprehensive features, and screen the multiple candidate responses using a preset scoring mechanism to obtain and output a final response.

Images