Multi-mode interaction intention understanding system of emotion accompanying robot

Abstract

The invention discloses a multi-modal interaction intention understanding system for an emotion accompanying robot, and relates to the technical field of robot accompanying, and the system comprises a modal input unit which obtains human body related information by means of a camera, the human body related information comprises a human face and a posture of a user, and obtains the voice of the user by using a microphone, and a feature extraction unit which is used for extracting features of the human body related information. The feature extraction unit carries out classification through feature extraction and information processing according to a behavior instruction output by a user and extracts intention features input in different modes, and the environment input unit carries out target detection on surrounding objects through an SSD algorithm and carries out scene judgment according to detected information to obtain a scene set. According to the method, through a multi-modal input layer, feature extraction and information processing, intention recognition, flexible association, feasibility judgment and man-machine cooperative interaction, more intentions can be expressed by using fewer instructions, and the error-tolerant rate and fluency in the man-machine interaction process are improved.

Description

Technical Field

[0001] This invention relates to the field of companion robot technology, specifically to a multimodal interaction intent understanding system for emotional companion robots. Background Technology

[0002] Chinese patent application number 202410635568.4 discloses "an emotional companion robot system, method, and emotional companion robot, comprising: a controller, used to receive user voice and user facial images, respiratory rate, and heart rate collected by a non-contact vital sign acquisition module, and convert the user voice into voice-text messages, and send all the above messages to a cloud server for processing, receive control commands returned by the cloud server after processing, and control the emotional companion robot to complete the corresponding actions after receiving the control commands, and the cloud server is used to receive information sent from the controller and judge the voice-text messages. If it is a command message, it generates a command that can control the robot's actions; otherwise, it analyzes the above information and feeds back the control commands obtained after analysis to the controller.

[0003] However, it is also important to achieve the association between different behaviors and a single intention in the same interactive context. Due to factors such as declining physical function, the elderly may not be able to accurately express their intentions through a single behavior. Therefore, using multiple behaviors and multiple channels can improve the system's fault tolerance and ensure the successful conduct of human-computer interaction. Different user behaviors and multiple modal behaviors can accurately express a certain meaning through the complementarity between intentions, which can improve the accuracy of the elderly's understanding of intentions. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal interaction intent understanding system for emotional companionship robots, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multimodal interaction intent understanding system for emotional companionship robots, comprising:

[0006] The modal input unit uses a camera to acquire human-related information, including the user's face and posture, and uses a microphone to acquire the user's voice.

[0007] The feature extraction unit, based on the user's output behavioral instructions, classifies them through feature extraction and information processing to extract the intent features of different modal inputs;

[0008] An environment input unit performs target detection on surrounding objects using the SSD algorithm and determines the scene based on the detected information to obtain a scene set.

[0009] The classification and detection unit divides facial gaze behavior into intentional gaze behavior and unintentional gaze behavior to detect intentional gaze behavior. It uses four features—gaze count, gaze content, gaze duration, and gaze movement speed—as input to the convolutional layer of the convolutional neural network and outputs intentional gaze behavior and unintentional gaze behavior through a fully connected layer.

[0010] A gaze intent acquisition unit performs gaze modeling on an intent sequence and an object sequence based on detected gaze intent behavior to obtain a gaze intent sequence.

[0011] The intent interaction unit obtains intent sets of various modalities by constructing a flexible association model, and performs flexible association through intent matching in the intent understanding database. It calculates the association probability between each behavior and each intent, and finally determines the final intent through the complementarity between intents, and applies it to the companion robot.

[0012] Preferably, the feature extraction unit includes a face extraction module, a body posture extraction module, and a voice extraction module;

[0013] The face extraction module extracts four gaze behavior features: gaze count, gaze content, gaze duration, and gaze movement speed.

[0014] The body posture extraction module first performs curve fitting, then calculates the Pearson correlation coefficient between each input body posture and each pre-collected instruction template, and saves the top three user intentions with the highest similarity.

[0015] The speech extraction module uses Baidu Voice's sentence analysis function to obtain a set of keywords and obtain user intent based on the arrangement of the keywords.

[0016] Preferably, the gaze intent acquisition unit obtains the gaze intent sequence by constructing a gaze model, and the construction process is as follows:

[0017] Step 1: Training dataset, including intent sequences and object sequences, where the intent sequence is a set of state variables, represented as follows: , where n represents the number of intentions, and the object sequence is the set of observable sequences, represented as , where m represents the number of objects;

[0018] Step 2: Estimate model parameters ;

[0019] First, calculate the probability of state change, i.e., the current state. Transition to state The probability of the state is defined. The number of samples is Probability of state change Represented as:

[0020] ,

[0021] Then, the state change probability matrix ;

[0022] Secondly, calculate the observation likelihood probability and define a certain state. And the corresponding object sequence The total number is The corresponding observation likelihood probability Represented as:

[0023] ,

[0024] Then, the observation likelihood probability matrix ;

[0025] Finally, the initial probability distribution is calculated, and the initial state is defined as follows. The number of data points at that time is Then the initial probability distribution Represented as:

[0026] ,

[0027] Then, the initial probability distribution vector is .

[0028] Preferably, in the gaze intent recognition process, the probability of the intent corresponding to each object and the node with the highest probability are recursively obtained based on the initial probability value sequence corresponding to the intent sequence. Finally, the intent with the highest probability corresponding to each object is calculated, thereby obtaining the intent sequence with the highest probability.

[0029] Preferably, the construction process of the intent understanding database is as follows: the intents in the intent understanding database consist of behavior, scene, semantics, and system instructions. Assume there are n intents in the intent understanding database. Each intent is composed of a sub-intent expressed by one of the four features: facial expression, body posture, scene, and voice. Then, the sub-intents of each sub-intent set are arranged according to... One of each method is taken and combined to obtain a fake user intent. Then, the obtained fake user intent is matched with the system-defined instruction set to finally determine an intent.

[0030] Preferably, the construction process of the flexible association model includes:

[0031] Step 1: Extract features from the input modal information, identify the intent features, and obtain sub-intent sets under four modalities: face, posture, scene, and voice. Calculate the probability corresponding to each sub-intent. ;

[0032] Step 2: Calculate and normalize the information entropy to obtain the entropy value corresponding to each mode. ;

[0033]

[0034] In the formula, This represents the sum of all intentions across all modalities. Indicates the calculation of probability Information entropy Indicates the sequence number.

[0035] Preferably, the construction process of the flexible association model further includes:

[0036] Step 3: Based on the obtained entropy value Calculate the contribution of each modality to intent recognition and determine the corresponding weight. ;

[0037]

[0038] In the formula, This indicates the number of intents within a given modality. This represents the sum of the number of intentions across all modalities. Indicates the number of input modes. It represents the probability of a certain intention in a certain modality within the overall scene;

[0039] Step 4: Randomly select one sub-intent from each of the four modalities' sub-intent sets, and then... By combining them in the order, a set of fake user intents is obtained. Then calculate the probability of the intention represented by each combination. The user intent set is formed by extracting the intent with the highest probability from three separate extractions.

[0040]

[0041] In the formula, This represents the user intent probability obtained through weighted calculation. It represents the probability of a certain intention in a certain modality within the overall scene. The weights represent a certain modality. This indicates the number of input modes.

[0042] Preferably, the construction process of the flexible association model further includes:

[0043] Step 5: Match the obtained user intent set with the intent understanding database. During the matching process, the system determines whether the intent matches the current scenario based on the target detection scenario. If the match is successful, the maximum value calculation is performed again, and the intent with the highest probability is taken as the final intent. If the match is unsuccessful, the system returns to the multimodal input layer.

[0044] Step 6: Check whether the current operation is in line with the task progress and determine whether the operation meets the human-machine safety execution standards. If it meets the execution standards, then execute it.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] This invention, through a multimodal input layer, feature extraction and information processing, intent recognition, flexible association, feasibility judgment, and human-computer collaborative interaction, not only enables the expression of more intents with fewer instructions, but also reduces the ambiguity of intents caused by non-standard behavior by complementing the intents between them. It also improves the fault tolerance and fluency of the human-computer interaction process. By complementing the sub-intent sets of multiple modalities and combining them with the current context, the invention calculates and understands the user's true intent and feeds the results back to the elderly. Through interaction, the intent is confirmed and corrected, and the user can perform corresponding operations to complete elderly care without guidance, thus meeting the user's emotional needs. Attached Figure Description

[0047] Figure 1 A schematic diagram of the overall system working principle provided for embodiments of the present invention;

[0048] Figure 2 This is a schematic diagram of the overall system provided in an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram illustrating the construction of the gaze model provided in an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram illustrating the construction of a flexible association model provided in an embodiment of the present invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Please see Figures 1 to 4 This invention provides a technical solution: a multimodal interaction intent understanding system for emotional companionship robots, comprising:

[0053] The modal input unit uses a camera to acquire human-related information, including the user's face and posture, and a microphone to acquire the user's voice.

[0054] The feature extraction unit, based on the user's output behavioral commands, classifies them through feature extraction and information processing to extract intent features from different modal inputs. This unit includes a face extraction module, a gesture extraction module, and a speech extraction module.

[0055] Face extraction module: Extracts four types of gaze behavior features: number of gazes, content of gaze, duration of gaze, and speed of gaze movement;

[0056] Body posture extraction module: First, curve fitting is performed, then the Pearson correlation coefficient between each input body posture and each pre-collected instruction template is calculated, and the top three user intentions with the highest similarity are saved.

[0057] Speech extraction module: Uses Baidu Voice's sentence analysis function to obtain a set of keywords and obtain user intent based on the arrangement of the keywords;

[0058] The environment input unit uses the SSD algorithm to detect surrounding objects and determines the scene based on the detected information to obtain a scene set.

[0059] The classification and detection unit divides facial gaze behavior into intentional gaze behavior and unintentional gaze behavior to detect intentional gaze behavior. It uses four features—gaze count, gaze content, gaze duration, and gaze movement speed—as input to the convolutional layer of the convolutional neural network, and outputs intentional gaze behavior and unintentional gaze behavior through a fully connected layer.

[0060] The gaze intent acquisition unit performs gaze modeling on the intent sequence and object sequence based on the detected gaze intent behavior to obtain the gaze intent sequence.

[0061] The intent interaction unit obtains intent sets of various modalities by constructing a flexible association model, and performs flexible association through intent matching in the intent understanding database. It calculates the association probability between each behavior and each intent, and finally determines the final intent through the complementarity between intents, and applies it to the companion robot.

[0062] The present invention will be further described below:

[0063] The gaze intent acquisition unit obtains a gaze intent sequence by constructing a gaze model;

[0064] The model consists of an initial probability distribution vector. State change probability matrix and observation likelihood probability matrix The model parameters Represented as: This includes two sets of variables: an unobservable sequence of random states and an observable sequence. The unobservable sequence of random states is determined by an initial probability distribution vector. and state change probability The observable sequence is determined by the observation likelihood probability matrix. Decide;

[0065] Will Defined as a set of state variables, Defined as a set of observable sequences, therefore, the state change probability matrix Represented as:

[0066] ,

[0067] in, Represented as: That is, in Moment State exist Shifting to a new state The probability of;

[0068] Observation likelihood probability matrix Represented as:

[0069] ,

[0070] in, Represented as: That is, in Moment State Generated under conditions The probability of;

[0071] Initial probability distribution vector Represented as:

[0072] ,

[0073] in, Represented as: That is, in The state at time is The probability of;

[0074] Furthermore, based on the intent sequence and object sequence within the above model framework, gaze modeling is constructed as follows:

[0075] Step 1: Training dataset, including intent sequences and object sequences, where the intent sequence is a set of state variables, represented as follows: , where n represents the number of intentions, and the object sequence is the set of observable sequences, represented as , where m represents the number of objects;

[0076] Step 2: Estimate model parameters ;

[0077] First, calculate the probability of state change, i.e., the current state. Transition to state The probability of the state is defined. The number of samples is Probability of state change Represented as:

[0078] ,

[0079] Then, the state change probability matrix ;

[0080] Secondly, calculate the observation likelihood probability and define a certain state. And the corresponding object sequence The total number is The corresponding observation likelihood probability Represented as:

[0081] ,

[0082] Then, the observation likelihood probability matrix ;

[0083] Finally, the initial probability distribution is calculated, and the initial state is defined as follows. The number of data points at that time is Then the initial probability distribution Represented as:

[0084] ,

[0085] Then, the initial probability distribution vector is ;

[0086] Step 3, when the object is At that time, the initial probability value sequence corresponding to the intention sequence is defined as follows: ,in, The formula is: Therefore, when the object sequence is input At that time, the probability of the intention corresponding to each object is obtained by recursion. and the node with the highest probability :

[0087] , ,

[0088] When the object is At that time, the probability value of the maximum intention for: Maximum probability value Corresponding Intent for: ;

[0089] Finally, the maximum probability of the intent corresponding to each object is calculated. :

[0090] ,

[0091] The intention sequence with the highest probability is: ;

[0092] The intent interaction unit obtains intent sets of various modalities by constructing a flexible association model, and flexibly associates them through intent matching in the intent understanding database. Therefore, the construction process of the intent understanding database is as follows:

[0093] The intents in the intent understanding database consist of behavior (facial expression + posture), context, semantics, and system commands. Assuming there are n intents in the database, each intent is composed of a sub-intent expressed by one of the four features (facial expression, posture, context, and speech). Taking posture as an example, when a user inputs a posture command, the system calculates the Pearson correlation coefficient between its fitted curve and the action template to obtain a set of action types for that posture. Then, the system filters intents based on factors such as the similarity between the posture command and various posture templates, ultimately obtaining n posture sub-intents from the posture sub-intent set. Similarly, other features also yield their respective sub-intent sets. Finally, the sub-intents of each sub-intent set are sorted according to... One of each sub-intent is taken and combined to obtain a pseudo user intent. Since the combination of the sub-intents may not be meaningful, the obtained pseudo user intent needs to be matched with the system-defined instruction set to finally determine an intent. Arranged according to different sub-intents, the system's intent understanding database is obtained.

[0094] The construction process of the flexible association model is as follows:

[0095] Step 1: Extract features from the input modal information, identify the intent features, and obtain sub-intent sets under four modalities: face, posture, scene, and voice. Calculate the probability corresponding to each sub-intent. ;

[0096] Step 2: Calculate and normalize the information entropy to obtain the entropy value corresponding to each mode. ;

[0097]

[0098] In the formula, This represents the sum of all intentions across all modalities. Indicates the calculation of probability Information entropy Indicates the sequence number of the intended action;

[0099] Step 3: Based on the obtained entropy value Calculate the contribution of each modality to intent recognition and determine the corresponding weight. ;

[0100]

[0101] In the formula, This indicates the number of intents within a given modality. This represents the sum of the number of intentions across all modalities. Indicates the number of input modes. It represents the probability of a certain intention in a certain modality within the overall scene;

[0102] Step 4: Randomly select one sub-intent from each of the four modalities' sub-intent sets, and then... By combining them in the order, a set of fake user intents is obtained. Then calculate the probability of the intention represented by each combination. The user intent set is formed by extracting the intent with the highest probability from three separate extractions.

[0103]

[0104] In the formula, This represents the user intent probability obtained through weighted calculation. It represents the probability of a certain intention in a certain modality within the overall scene. The weights represent a certain modality. Indicates the number of input modes;

[0105] Step 5: Match the obtained user intent set with the intent understanding database. During the matching process, the system determines whether the intent matches the current scenario based on the target detection scenario. If the match is successful, the maximum value calculation is performed again, and the intent with the highest probability is taken as the final intent. If the match is unsuccessful, the system returns to the multimodal input layer.

[0106] Step 6: Check whether the current operation is in line with the task progress and determine whether the operation meets the human-machine safety execution standards. If it meets the execution standards, then execute it.

[0107] Working Principle: In use, this invention employs a modal input unit to acquire human-related information, including the user's face and posture, via a camera. Simultaneously, a microphone captures the user's voice. A feature extraction unit classifies the user's output behavioral commands through feature extraction and information processing, extracting the intent features of different modal inputs. An environmental input unit uses the SSD algorithm to detect surrounding objects and determines the scene based on the detected information, resulting in a scene set. A classification detection unit categorizes facial gaze behavior into intentional and unintentional gaze behaviors for the detection of intentional gaze behavior. The number of gazes is used to... Four features—gaze content, gaze duration, and gaze movement speed—are used as inputs to the convolutional layers of a convolutional neural network. Fully connected layers output intentional and unintentional gaze behaviors. A gaze intent acquisition unit models the intent sequence and object sequence based on the detected intentional gaze behaviors to obtain a gaze intent sequence. An intent interaction unit constructs a flexible association model to obtain intent sets of various modalities and performs flexible association through intent matching in an intent understanding database. It calculates the association probability between each behavior and each intent and ultimately determines the final intent through the complementarity of intents, which is then applied to the companion robot for emotional support.

[0108] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0109] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An emotional companion robot multimodal interaction intent understanding system, characterized in that, The application relates to a multi-modal input system for a companion robot. The system comprises a modal input unit, a feature extraction unit, an environment input unit, a classification detection unit, a gaze intention obtaining unit and an intention interaction unit. The modal input unit acquires human-related information, including a user's face, body posture and voice, by means of a camera and a microphone. The feature extraction unit extracts intention features of different modal inputs by feature extraction and information processing according to a user's output behavior instruction. The environment input unit detects surrounding objects by an SSD algorithm and determines a scene according to the detected information to obtain a scene set. The classification detection unit divides a face gaze behavior into an intentional gaze behavior and an unintentional gaze behavior to realize detection of the intentional gaze behavior. The gaze intention obtaining unit models a gaze according to the detected intentional gaze behavior, obtains an intention sequence and an object sequence, and obtains a gaze intention sequence. 2.The emotional companion robot multimodal interaction intent understanding system of claim 1, wherein: The intention interaction unit obtains intention sets of various modalities by constructing a flexible association model, flexibly associates the intention sets by intention matching of an intention understanding database, calculates association probabilities between each behavior and each intention, and finally determines a final intention by complementation between intentions and applies the final intention to the companion robot. The feature extraction unit comprises a face extraction module, a body posture extraction module and a voice extraction module. The face extraction module extracts four gaze behavior features, including a gaze frequency, a gaze content, a time length of the gaze content and a line-of-sight moving speed. The body posture extraction module firstly performs curve fitting, then calculates a Pearson correlation coefficient of each input body posture and each pre-collected instruction template, and saves user intentions with the top three similarity degrees. 3.The emotional companion robot multimodal interaction intent understanding system of claim 1, wherein: The voice extraction module uses a sentence analysis function of Baidu voice to obtain a keyword set and acquires user intentions according to arrangement of the keywords. Step one, training data set, including intent sequence and object sequence, wherein the intent sequence is a state variable set, denoted as , n represents the number of intents, and the object sequence is a set of observable sequences, denoted as , wherein m represents the number of objects; Step two, estimating model parameters ; First, the state change probability, i.e. the current state is calculated The probability of moving to state is defined as the number of samples of state , and the state change probability is expressed as: ， So, the state change probability matrix ; Secondly, calculate the observation likelihood probability and define a certain state. And the corresponding object sequence The total number is The corresponding observation likelihood probability Represented as: ， So, the observation likelihood probability matrix ; Finally, the initial probability distribution is calculated, defining the initial state as the number of data at time the initial probability distribution is expressed as: ， So, the initial probability distribution vector is . 4.The emotional companion robot multimodal interaction intent understanding system of claim 3, wherein: The gaze intention obtaining unit obtains a gaze intention sequence by constructing a gaze model, and the construction process is as follows. 5.The emotional companion robot multimodal interaction intent understanding system of claim 1, wherein: The construction process of the intention understanding database is as follows: the intention in the intention understanding database is composed of behaviors, scenes, semantics and system instructions, there are n intentions in the intention understanding database, any intention is composed of a sub-intention expressed by each of the four features of face, body posture, scene and voice, then one sub-intention of each sub-intention set is taken out in a manner of combination to obtain a false user intention, then the obtained false user intention is matched with a system defined instruction set, and finally an intention is determined. 6.The emotional companion robot multimodal interaction intent understanding system of claim 1, wherein: In the gaze intention recognition process, a probability and a maximum probability node of an intention corresponding to each object are obtained according to a sequence of initial probability values corresponding to the intention sequence, and finally a maximum probability intention corresponding to each object is obtained, and a maximum-probability intention sequence is obtained. Step one, feature extraction is performed on the input individual modal information, and recognition is performed according to the intention features, to obtain a sub-intention set under four modalities of face, body posture, scene and voice, and to obtain a probability corresponding to each sub-intention ; Step two, information entropy calculation and normalization processing are carried out to obtain the entropy value corresponding to each mode ; wherein, represents the sum of all intents in all modalities, represents the information entropy of the probability of computing the probability represents the intent serial number. 7.The emotional companion robot multimodal interaction intent understanding system of claim 6, wherein: The construction process of the flexible association model comprises the following steps. Step three, calculating the contribution value of each modality to the intent recognition Calculate the contribution value of each modality to the intent recognition ; wherein, denotes the number of intents in a certain modality intent, denotes the number of all intents in all modalities and, denotes the number of input modalities, denotes the probability of a certain intent in a certain modality in the overall scenario; Step four, randomly select one sub-intent from each of the four modal sub-intent sets, and combine them in the order of to obtain a set of false user intents , then calculate the probability of the intent represented by each combination , and form a user intent set by taking the three intents with the highest probabilities. In the formula, represents the probability of user intention calculated by weighting, represents the probability of a certain intention in the overall scene under a certain modality, represents the weight of a certain modality, represents the number of input modalities. 8.The emotional companion robot multimodal interaction intent understanding system of claim 7, wherein: The construction process of the flexible association model further comprises the following steps. The construction process of the flexible association model further comprises the following steps. Step five: The obtained user intention set is matched with an intention understanding database, in the matching process, the system determines whether the intention conforms to a current scene according to scene determination of target detection, if the matching is successful, a maximum value is calculated again, a maximum-probability intention is taken as a final intention, and if the matching is unsuccessful, the multi-modal input layer is returned. Step six: Whether a current operation conforms to a task progress is inquired, and whether the operation conforms to a man-machine safety execution standard is determined, if the operation conforms to the execution standard, the operation is executed.

Images