Intention understanding personalized man-machine interaction method based on multimode fusion

Abstract

The invention relates to an intention understanding personalized man-machine interaction method based on multi-mode fusion, which is characterized in that weights are set for staring, voice and gesture information sources to cater to user preferences, in an information fusion process, a pre-fusion information source is obtained by weighting the three information sources, the four information sources are combined and fused to estimate user intentions, and the intention understanding personalized man-machine interaction method based on multi-mode fusion is realized. The user feeds back the result, records whether the result accords with the expectation of the user or not, updates the modal weight according to the prediction results after the specified number of times is predicted every time, and sends an effective intention instruction to the robot, and the robot completes a corresponding task, compared with the prior art, the method has the advantages of high accuracy and high reliability, and the method is suitable for popularization and application. In addition, the method has the advantages that the weight can be periodically adjusted according to user feedback, user preferences can be catered, in a specific scene, the method can adapt to various modes of different scenes by adjusting the weight, the application scene of man-machine interaction is improved, and personalized man-machine interaction experience is provided.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-computer interaction, and in particular, relates to a personalized human-computer interaction method based on intention understanding of multi-modal fusion. Background Art

[0002] Most of the existing human-computer interaction technologies are single-mode, with a relatively simple interaction method. The interaction process is heavily dependent on the user's attention, requiring the user to concentrate, and the interaction results are easily affected by environmental factors, which is not suitable for complex and changeable life scenes. Multimodal human-computer interaction can provide users with multiple interaction methods, and the number of interaction methods that users can choose from has increased. Users have different usage habits for each mode, and the current multimodal information fusion method has the problems of poor personalization and low reliability, which makes it difficult to meet user preferences and has great limitations. It is necessary to propose a more personalized and reliable multimodal information fusion method. Summary of the invention

[0003] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a personalized human-computer interaction method based on multi-modal fusion intention understanding.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] The present invention provides a personalized human-computer interaction method based on intention understanding of multi-modal fusion, comprising the following steps:

[0006] Use multiple sensors to collect the user's gaze modality data, gesture modality data, and voice modality data at the same time;

[0007] The three modal data are processed in parallel to obtain the labels and corresponding accuracy of each of the three modalities;

[0008] The labels of the three modalities are taken as propositions of the evidence theory, the accuracy of each label is taken as the confidence of its proposition, and the set of propositions of each modality and its confidence is taken as each evidence source;

[0009] According to prior knowledge, an initial weight value is set for each evidence source, and the three evidence sources are pre-fused according to the weight to obtain a pre-fused evidence source;

[0010] The pre-fusion evidence source, gaze evidence source, gesture evidence source and voice evidence source are fused using the improved evidence theory to obtain the final evidence source, and the proposition with the highest confidence is selected from the final evidence source as the final intention recognition result;

[0011] The machine takes corresponding actions based on the final recognition intention results.

[0012] Furthermore, the gaze modality data is obtained through an eye tracker worn by the user, including a user scene video stream and a user line of sight coordinates.

[0013] Furthermore, the gesture modality data is acquired through a data glove worn by the user, including 18 hand joint angle data.

[0014] Furthermore, the speech modality data is acquired through a microphone array on the machine, including: time series audio data.

[0015] Furthermore, the gaze modality data processing process includes: carrying the yolov5 algorithm framework on the scene video stream of the gaze modality data to obtain a target recognition result, wherein the target recognition result includes a label and its accuracy, and determining the label of the gaze modality and its corresponding accuracy through the target recognition result and the positional relationship between the gaze point coordinates and the target recognition frame.

[0016] Furthermore, the gesture modality data processing process includes: using a random forest method to vote on the hand joint angle data to obtain the number of votes corresponding to each label, and then obtaining the gesture modality label and its corresponding accuracy through normalization.

[0017] Furthermore, the speech modality data processing process includes: preprocessing and feature extracting the time series audio data, using a bidirectional LSTM neural network for recognition, and obtaining corresponding labels and their corresponding accuracy rates.

[0018] Furthermore, the three evidence sources are pre-fused according to the weights to obtain a pre-fused evidence source, and the pre-fusion formula is:

[0019] m f (A i )=w e ·m e (A i )+w g ·m g (A i )+w v ·m v (A i )

[0020] Among them A i Represents proposition i, m e (A i ) represents the confidence of the gaze evidence source proposition i, m g (A i ) represents the confidence of the gesture evidence source proposition i, m v (A i ) represents the confidence of the speech evidence source proposition i, w erepresents the weight of the gaze evidence source, w g represents the weight of the gesture evidence source, w v represents the weight of the speech evidence source, m f (A i ) represents the confidence of the pre-fused evidence source proposition i.

[0021] Furthermore, the improved evidence theory is used to fuse the pre-fused evidence source, gaze evidence source, gesture evidence source and voice evidence source to obtain the final evidence source. The specific fusion formula is:

[0022]

[0023] Among them, a, b, c, d are the question numbers, A j For Proposition A a , A b , A c , A d The intersection proposition of For Proposition A j confidence level.

[0024] Furthermore, the weight is updated periodically, and each time the first preset number of intention recognitions is performed is a cycle, and the weight update formula is:

[0025]

[0026] Among them, c e The number of times the gaze mode is valid in a single cycle, c g is the number of times the gesture mode is valid in a single cycle, c v is the number of times the speech mode is valid in a single cycle, k = e, g, v, w k Indicates the weight of a modal evidence source;

[0027] The number of times the modality is valid in a single cycle refers to the source of evidence obtained after processing a certain modal data, and the proposition with the highest confidence is the same as the user's true intention. Within a cycle, the same number of times is the number of times the modality is valid in a single cycle.

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] (1) Compared with the existing single-modal interaction technology, the technical solution proposed in the present invention has rich interaction modes, and can realize multi-modal information fusion human-computer interaction based on voice, gaze and gesture, thereby improving the reliability and naturalness of human-computer interaction.

[0030] (2) Compared with the traditional evidence theory algorithm, the improved evidence theory algorithm adopted by the present invention sets weights for each channel to perform information fusion, which has the advantages of high accuracy and high reliability. In addition, the present invention can periodically adjust the weights according to user feedback, which can cater to user preferences. In specific scenarios, the weights can be adjusted to adapt to different modes in different scenarios, thereby improving the application scenarios of human-computer interaction and providing a personalized human-computer interaction experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a diagram of the multimodal information fusion architecture of the present invention;

[0032] Figure 2 This is a flowchart of the gaze information processing process of the present invention;

[0033] Figure 3 This is a flowchart of the gesture information processing process of the present invention;

[0034] Figure 4 Improved evidence theory framework diagram for the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0036] Embodiment 1:

[0037] The purpose of the present invention is to address the deficiencies of the existing technology and propose a personalized human-computer interaction method based on multi-modal fusion intention understanding, which is used to provide personalized human-computer interaction services. Based on the feedback-based weighted DS evidence theory, an intention reasoning method that integrates three modalities of gaze, voice, and gesture is designed to control the robot system and achieve highly reliable and personalized human-computer interaction.

[0038] To achieve the above purpose, the concept of the present invention is: set three parallel threads to receive and process gaze, voice and gesture information respectively, each thread runs in parallel and outputs a separate result, including recognition label and probability. Then the output of the three threads is used as the input of information fusion, and the improved DS evidence theory is used to perform information fusion to obtain the final instruction. The user gives feedback based on the final result, records whether the operation meets the user's expectations, and then updates the weight of each modality after a certain number of interactions. Repeat the above steps, periodically update the weight, and realize reliable and personalized human-computer interaction.

[0039] According to the above concept, the present invention is implemented by the following technical solutions:

[0040] A personalized human-computer interaction method for intention understanding based on multi-modal fusion is characterized in that weights are set for the three information sources of gaze, voice and gesture to cater to user preferences. In the information fusion process, the pre-fusion information source is obtained by weighting the three information sources, and the four information sources are combined and fused to estimate the user's intention. The user gives feedback on the result and records whether the result meets the user's expectations. After each specified number of predictions, the modal weights are updated according to these prediction results, and the effective intention instructions are sent to the robot, and the robot completes the corresponding task, including the following steps:

[0041] (1) Information collection: Use multiple sensors to collect data from various modalities at the same time to provide a data basis for subsequent information fusion. Gaze modal data uses an eye tracker to collect the user's visual field scene video stream and user gaze information including gaze point coordinates; gesture modal data uses data gloves to collect the user's hand joint angles, including the bending angles of each finger joint; voice modal data uses a voice module to collect user voice information. Three threads are completed in parallel.

[0042] (2) Information processing: The collected data is processed in parallel using deep learning or machine learning methods to obtain independent intention recognition results and accuracy rates, providing a source of evidence for information fusion. For gaze data processing, the yolov5 algorithm framework is installed on the scene video stream to determine the label and accuracy of the object the user is gazing at through the target recognition result and the positional relationship between the gaze point coordinates and the target recognition frame; for gesture data processing, the random forest method is used to vote on the hand joint angle data to obtain the number of votes corresponding to each label, and then the recognition result, including the label and accuracy, is obtained after normalization. For voice modality data, the microphone array on the machine is used to obtain time series audio data, and the time series audio data is preprocessed and feature extracted, and then recognized using a bidirectional LSTM neural network to obtain the corresponding label and its corresponding accuracy. Three threads run in parallel.

[0043] (3) Information fusion: The three modal information after receiving information processing, including the recognition accuracy of each tag, is fused using the feedback weighted evidence theory to obtain the final recognition result and accuracy.

[0044] The robot completes the task: execute effective intention instructions. It grabs the target object through traditional vision-based methods and places it in the specified location. This content is relatively mature and will not be introduced here.

[0045] Next, we will introduce the improved evidence theory algorithm

[0046] Basic definition: A non-empty set E contains two mutually exclusive events, called an identification frame. Its power set 2E is the set of all subsets, including itself and the empty set. Each subset in the power set becomes a focal element, and each focal element corresponds to a value between [0, 1], which is a measure of the confidence of the evidence. The basic probability assignment (BPA) on the identification frame is also called the mass function, which satisfies:

[0047]

[0048] Among them, m(A) is the confidence of proposition A.

[0049] In addition, there are the belief function Bel(A) and the likelihood function pl(A). Bel(A) represents the minimum degree of support for proposition A, and pl(A) represents the degree to which the evidence does not oppose proposition A. They are defined as follows:

[0050]

[0051] Weighted combination rule: The combination rule combines several independent quality functions. The weighted combination rule first assigns a weight w to each source of evidence. i , and then weight the basic probability distribution function.

[0052] m f (A i )=w e ·m e (A i )+w g ·m g (A i )+w v ·m v (A i )

[0053] Among them A i Represents proposition i, m e (A i ) represents the confidence of the gaze evidence source proposition i, m g (A i ) represents the confidence of the gesture evidence source proposition i, m v (A i ) represents the confidence of the speech evidence source proposition i, w e represents the weight of the gaze evidence source, w g represents the weight of the gesture evidence source, w v represents the weight of the speech evidence source, m f (A i ) represents the confidence of the pre-fused evidence source proposition i.

[0054] Then, multiple evidence sources are synthesized through DS combination rules.

[0055]

[0056] Among them, a, b, c, d are the question numbers, A j For Proposition A a , A b , A c , A d The intersection proposition of For Proposition A j confidence level.

[0057] Weight update, the present invention considers as follows: first, weights of the three modes are initialized according to prior knowledge w e =0.4, w g =0.3, w v = 0.3, and then update the channel weight according to user feedback. When the user feedback result is positive, it means that the intent understanding estimate is correct, indicating that it is consistent with the final intent. Figure 1 The modality that is consistent with the final intention plays a positive role, while the modality that is inconsistent with the final intention plays a negative role. Figure 1 When the user feedback is negative, it means that the intent understanding estimation is wrong, which means that the modality with the consistent intent and the final result has a reverse effect, while the effect of the modality with inconsistent intent and the final result is too small. Therefore, the weight of the modality inconsistent with the final intent should be increased, and the weight of the modality inconsistent with the final intent should be reduced. Figure 1 After every ten attempts of intent estimation, the weight of each modality and the user’s true intent are counted. Figure 1 The more correct times, the greater the weight should be, and it is updated periodically. The weight w update method for a certain mode is as follows:

[0058]

[0059] Among them, c e The number of times the gaze mode is valid in a single cycle, c g is the number of times the gesture mode is valid in a single cycle, c v is the number of times the speech mode is valid in a single cycle, k = e, g, v, w k Indicates the weight of a modal evidence source;

[0060] The number of times the modality is valid in a single cycle refers to the source of evidence obtained after processing a certain modal data, and the proposition with the highest confidence is the same as the user's true intention. Within a cycle, the same number of times is the number of times the modality is valid in a single cycle.

[0061] Embodiment 2:

[0062] This embodiment should be understood to include at least all the features of any one of the aforementioned embodiments and to be further improved thereon.

[0063] A preferred embodiment of the present invention is described in detail as follows: Figure 1 , a personalized human-computer interaction method based on intention understanding of multimodal fusion uses an eye tracker, data gloves and microphone array to collect the user's gaze, gesture and voice information, and uses deep learning and machine learning methods to identify the modal data to obtain a separate intention result. The intention results of the three modalities are fused using the improved evidence theory to obtain the fused intention result. The final intention instruction is determined based on user feedback, and the intention instruction is sent to the robotic arm system, which executes the intention instruction. The human-computer interaction solution with multimodal information fusion is applied to the task of grasping target objects in multiple scenarios.

[0064] Background: A common meeting scenario is selected here. Drinks, bananas, snacks, etc. are placed on the table. The user wants to drink water during the meeting, but cannot speak loudly and it is not convenient to use voice for human-computer interaction. However, the user can use gestures and gaze to interact with the robotic arm to complete the grabbing of the drink.

[0065] (1) Modal information collection: The user wears an eye tracker and data gloves to obtain gaze information I e and gesture information I g . Among them I e Including the user scene video stream Image and the user's line of sight coordinates Gaze, I g Includes 18 hand joint angle data Angles.

[0066] (2) Modal information recognition: for gaze data I e processing, such as Figure 2 , the yolov5 algorithm framework is installed on the scene video stream Image, and the gaze evidence source S is determined by the target recognition result (label and accuracy) and the positional relationship between the gaze point coordinate Gaze and the target recognition frame. e , including each label and its confidence; for gesture data I g Processing, such as Figure 3 , the random forest method is used to vote on the hand joint angle data Angles, and the number of votes corresponding to each label is obtained. Then, after normalization, the gesture evidence source S is obtained. g , including each label and confidence. The label and accuracy here correspond to the proposition A of the evidence theory and its quality function m(A). A1, A2, A3 represent beverages, bananas, and snacks respectively; m(A1), m(A2), m(A3). are the confidences of beverages, bananas, and snacks respectively.

[0067] (3) Evidence source preprocessing: weighting the evidence source to obtain the pre-fused evidence source S f . Includes propositions A1, A2, A3 and m f (A1), m f (A2), m f (A3). The specific formula is as follows:

[0068] m f (A i )=w e ·m e (A i )+w g ·m g (A i ) (10)

[0069] where w e 、w g are the weights of gaze and gesture respectively, i=1, 2, 3.

[0070] (4) Information fusion: Figure 4 As shown, the pre-fused evidence source S is transformed into f , stare at the source of evidence S e and gesture evidence source S g After fusion, the proposition with the highest confidence is selected as the final intention result. The specific fusion formula is as follows:

[0071]

[0072]

[0073] Among them, U, V, W are A1, A2, A3, and i=1, 2, 3.

[0074] Existing gaze evidence source S e and gesture evidence source S g . S e is m e (A1) = 0.8, m e (A2) = 0.1, m e (A3) = 0.1, S g is m g (A1) = 0.1, m g (A2) = 0.8, m g (A3) = 0.1, the user prefers to select the target object by looking at it, and w e =0.7, w g =0.3, w v =0, then S f is m f (A1)=0.59,m f(A2) = 0.31, m f (A3)=0.1.

[0075] 1-K=0.8×0.1×0.59+0.1×0.8×0.31+0.1×0.1×0.1=0.073

[0076]

[0077] The proposition with the highest confidence is selected as the final intent result. The final intent result is A1, which means beverage, which meets the user's expectations.

[0078] For weight update, after every ten attempts of information fusion to understand the intention, the weight of each modality is calculated and updated according to the weight update formula.

[0079] (5) Robotic arm movement: Taking the beverage is used as the intention instruction of the robotic arm system. The robotic arm recognizes the beverage based on the visual detector, calculates the grasping posture, and then grasps the beverage and places it next to the user's hand.

[0080] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A personalized human-computer interaction method based on intention understanding of multi-modal fusion, characterized in that: The following steps are involved: Use multiple sensors to collect the user's gaze modality data, gesture modality data, and voice modality data at the same time; The three modal data are processed in parallel to obtain the labels and corresponding accuracy of each of the three modalities; The labels of the three modalities are taken as propositions of the evidence theory, the accuracy of each label is taken as the confidence of its proposition, and the set of propositions of each modality and its confidence is taken as each evidence source; According to prior knowledge, an initial weight value is set for each evidence source, and the three evidence sources are pre-fused according to the weight to obtain a pre-fused evidence source; The pre-fusion evidence source, gaze evidence source, gesture evidence source and voice evidence source are fused using the improved evidence theory to obtain the final evidence source, and the proposition with the highest confidence is selected from the final evidence source as the final intention recognition result; The machine takes corresponding actions based on the final recognition intention results.

2. According to claim 1, a personalized human-computer interaction method based on intention understanding of multi-modal fusion is characterized in that: The gaze modality data is obtained through an eye tracker worn by the user, including a user scene video stream and a user line of sight coordinates.

3. According to claim 1, a personalized human-computer interaction method based on intention understanding of multi-modal fusion is characterized in that: The gesture modality data is obtained through a data glove worn by the user, including 18 hand joint angle data.

4. According to claim 1, a personalized human-computer interaction method based on intention understanding of multi-modal fusion is characterized in that: The speech modality data is acquired through a microphone array on the machine, including: time series audio data.

5. According to claim 1, a personalized human-computer interaction method based on intention understanding of multi-modal fusion is characterized in that: The gaze modality data processing process includes: carrying the yolov5 algorithm framework on the scene video stream of the gaze modality data to obtain a target recognition result, wherein the target recognition result includes a label and its accuracy, and determining the label of the gaze modality and its corresponding accuracy through the target recognition result and the positional relationship between the gaze point coordinates and the target recognition frame.

6. According to claim 1, a personalized human-computer interaction method based on intention understanding of multi-modal fusion is characterized in that: The gesture modality data processing process includes: using a random forest method to vote on the hand joint angle data to obtain the number of votes corresponding to each label, and then obtaining the gesture modality label and its corresponding accuracy through normalization processing.

7. According to claim 1, a personalized human-computer interaction method based on intention understanding of multi-modal fusion is characterized in that: The speech modality data processing process includes: preprocessing and feature extraction of the time series audio data, using a bidirectional LSTM neural network for recognition, and obtaining corresponding labels and their corresponding accuracy rates.

8. According to claim 1, a personalized human-computer interaction method based on intention understanding of multi-modal fusion is characterized in that: The three evidence sources are pre-fused according to the weights to obtain a pre-fused evidence source. The pre-fusion formula is: m f (A i )=w e ·m e (A i )+w g ·m g (A i )+w v ·m v (A i ) Among them A i Represents proposition i, m e (A i ) represents the confidence of the gaze evidence source proposition i, m g (A i ) represents the confidence of the gesture evidence source proposition i, m v (A i ) represents the confidence of the speech evidence source proposition i, w e represents the weight of the gaze evidence source, w g represents the weight of the gesture evidence source, w v represents the weight of the speech evidence source, m f (A i ) represents the confidence of the pre-fused evidence source proposition i.

9. The personalized human-computer interaction method based on intention understanding of multi-modal fusion according to claim 1 is characterized in that: The improved evidence theory is used to fuse the pre-fusion evidence source, gaze evidence source, gesture evidence source and voice evidence source to obtain the final evidence source. The specific fusion formula is: Among them, a, b, c, d are the question numbers, A j For Proposition A a , A b , A c , A d The intersection proposition of For Proposition A j confidence level.

10. The personalized human-computer interaction method based on intention understanding of multi-modal fusion according to claim 1, characterized in that: The weight is updated periodically, and each first preset number of intention recognitions is considered a cycle. The weight update formula is: Among them, c e The number of times the gaze mode is valid in a single cycle, c g is the number of times the gesture mode is valid in a single cycle, c v is the number of times the speech mode is valid in a single cycle, k = e, g, v, w k Indicates the weight of a modal evidence source; The number of times the modality is valid in a single cycle refers to the source of evidence obtained after processing a certain modal data, and the proposition with the highest confidence is the same as the user's true intention. Within a cycle, the same number of times is the number of times the modality is valid in a single cycle.

Images