Multi-modal fusion control method for man-machine interaction system

Abstract

The invention discloses a multi-mode fusion control method for a man-machine interaction system. Firstly, a reinforcement learning algorithm is developed to enable the robot to learn an optimal modal priority strategy by taking human control of the mobile robot as an example; in addition, a multi-modal fusion technology is innovated based on a null-space behavior control (NSBC) method, and after a reinforcement learning algorithm obtains a specific modal priority, the method can seamlessly integrate potential conflict control instructions from multiple modals. According to the NSBC method, the expansion from synchronous operation to asynchronous operation is realized, so that the NSBC method is suitable for a multi-mode HRI system. In order to verify the effectiveness of the framework method, a movement control experiment of the mobile robot in the obstacle environment is carried out. Results show that the method is superior to a traditional single-mode HRI control method and a fixed weighted multi-mode method in the aspects of average task completion time, collision rate, artificial participation degree, task error rate and the like. The questionnaire survey at the last stage of the experiment further verifies the acceptance of the user to the HRI framework provided by the user.

Description

Technical Field

[0001] The present invention relates to the field of human-machine hybrid enhanced intelligence technology, and in particular to a multimodal fusion control method for a human-machine interaction system. Background Art

[0002] With the rapid development of artificial intelligence technology, it has become simpler and more convenient for people to obtain various types of data (text, images, voice, etc.), and multimodal technology has become the next hot spot. Multimodal technology can be applied to various fields, such as human-computer interaction, education and training, deep learning, visual tracking, and so on. In human-computer interaction (HCI), multimodal technology can provide a more natural and intuitive way of interaction. For example, by combining voice and gesture recognition, communication with machines is more intuitive and convenient. At the same time, the emerging human-computer interaction modalities (audio, vision, touch, kinesthetic, etc.) are more usable than the traditional human-computer interaction model based on PC using keyboard and mouse.

[0003] In existing research, human-computer interaction (HCI) is defined as the technical field that studies the interaction between humans and computers and their mutual influence. When the "machine" in HCI expands to include intelligent "robots," this form of interaction transcends the scope of traditional HCI and evolves into human-robot interaction (HRI), opening up new avenues for HCI research.

[0004] Currently, human-robot interaction modalities (HRIs) exhibit significant multimodal and cross-modal characteristics. However, the naturalness, efficiency, and stability of multimodal and cross-modal interactions between humans and robots remain to be improved. The fusion of multimodal HRIs can eliminate the limitations and constraints inherent in traditional HRIs and fully leverage the communication, perception, and cognitive abilities of humans within HRIs, thereby enhancing the performance and user experience of HRI systems. Specifically, the fusion of multimodal HRIs offers several advantages: First, it improves the robustness and stability of HRI systems, making the interaction process more reliable; second, the fusion of multimodal HRIs meets the needs of a wide range of users, making human-robot interactions more natural and intuitive; third, it enhances task execution efficiency by allowing users to flexibly select and cross-use multiple input modes, such as voice, gesture, and multi-touch, thereby improving interaction efficiency; finally, the fusion of multimodal HRIs can improve the accuracy of user data input, further enhancing the robustness and stability of human-robot interactions.

[0005] Despite the numerous advantages of multimodal HRI fusion, a clear collaborative fusion framework for integrating various modes to achieve multimodal interaction is still lacking. Therefore, for multimodal HRI systems, exploring intelligent fusion methods for multimodal interaction between humans and robots is particularly important. Building on traditional single-modal HRI, this paper conducts in-depth research on the fusion mechanism of multiple HRI modalities, aiming to design an efficient multimodal intelligent fusion method for human-robot interaction within a multimodal HRI system, in order to promote the development of HRI technology and achieve a more natural, efficient, and stable HRI experience. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a multimodal fusion control method for human-computer interaction systems. A multimodal HRI system is constructed based on the traditional single HRI mode. At the same time, the modal priority is trained offline through reinforcement learning, and then the NSBC method is used to fuse different HRI modes, so as to construct a multimodal HRI fusion system in actual use.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a multimodal fusion control method for a human-computer interaction system, comprising the following steps:

[0008] Step 1: Implementation of multimodal HRI basic technology;

[0009] Step 2: Modal priority strategy learning based on reinforcement learning. Reinforcement learning modal priority supervision is designed by combining the deep Q-network algorithm (DQN) in reinforcement learning. The HRI system autonomously learns the modal priority strategy based on safety and efficiency criteria. In a simulated environment, the HRI system comprehensively considers obstacle distance, modal delay, recognition accuracy, light intensity, and noise level indicators to train the DQN.

[0010] Step 3: Use the NSBC method to fuse multi-modal HRI control instructions: project the low-priority modal instructions to the null space of the high-priority modal instructions.

[0011] In a preferred embodiment, step 1 includes the implementation of speech recognition, the implementation of gesture recognition, and communication; speech recognition is performed using a speech recognition module, gesture recognition is performed through a gesture recognition module, and a keyboard of a host computer is used to form a multimodal HRI, and corresponding recognition results are generated as control instructions.

[0012] In a preferred embodiment, in step 2, different HRI modal control instructions are used as the action set of the DQN algorithm, and a suitable mobile robot state set, such as the position of the mobile robot, is selected as the state set of the DQN algorithm. A reasonable reward function is designed to train the DQN by comprehensively considering obstacle distance, modal delay, recognition accuracy, light intensity and noise level indicators.

[0013] In a preferred embodiment, in the context of multimodal HRI, the state space X of each HRI mode is M And the control instruction u corresponding to the user i Defined as:

[0014]

[0015] Where q is the dimension of each mode state space, N is the number of HRI modes, and t n =nT represents the time after n sampling times, where T is the sampling time;

[0016] The dimension of the HRI modal state space is q = 4, and we have:

[0017] u i =[x i ,y i ,ω i ,α i ] (2)

[0018] Among them, x i 、y i 、ω i and α i They represent the longitudinal velocity, lateral velocity, angular velocity and absolute velocity of the i-th HRI modal control command respectively;

[0019] In a given environment Ξ, the state set S = {s t} represents various environmental conditions, where each state s t =[V o ,L env ,D obstacle ]∈R 3 Contains the noise level V perceived by the robot o , ambient light intensity L env and the distance D to the nearest obstacle obstacle ; Action set B = {b t}Contains the priority of the modality, and the mobile robot receives control commands from humans through the HRI modality.

[0020] In a preferred embodiment, the training process of the Deep Q Network algorithm DQN is as follows:

[0021] 1) State and action selection: At time t, the robot is in state s t When, according to a decay factor γ ε The regulated ε-greedy policy selects an action b t ; The ε-greedy strategy stipulates that the robot The probability of randomly selecting an action is The probability of selecting the action with the highest Q value is E Indicates the number of training rounds completed so far;

[0022] 2) State transfer and reward: Once the robot performs action b t , the robot will receive an action execution reward R t and transfer to the new state s t+1 ;

[0023] 3) Experience storage: experience tuple (s t ,b t ,R t ,s t+1 ) are then stored in an experience replay buffer called Ω; this buffer acts as a memory of past experiences that are reused during training, stabilizing the learning process and improving data efficiency;

[0024] 4) Offline training: Training proceeds to the next time step t+1, and the process continues iteratively. During offline training, experiences are randomly drawn from Ω to update the deep Q network algorithm DQN.

[0025] 5) Iteration: This process continues until training is complete, ensuring that the robot learns an optimal strategy for selecting actions based on the state of the environment to maximize its cumulative reward.

[0026] 6) Deterministic strategy: The goal of the deep Q network algorithm DQN is to obtain a deterministic strategy, expressed as π(b t ∣s t ), the deterministic strategy optimizes the total reward accumulated by the robot; after offline training is completed, the deep Q network algorithm DQN will generate a deterministic strategy π(b t ∣s t ); It should be emphasized that for a specific state s t ,π(b t ∣s t ) will produce a certain action b t , rather than a probability distribution over all possible actions.

[0027] In a preferred embodiment, the HRI modality priority algorithm includes training the network Q(s, b; W Q ,W V ,W A ) and a target network, the training network is used to update its parameters to reflect the changes in the HRI modality priority strategy and to select the next action; the target network provides a stable target Q value, which is used to calculate the loss function during training; the Q value function of the target network is The Q-value function provides the robot with a quantitative indicator of the quality of performing a specific action in a specific state by accumulating future rewards; a higher Q-value indicates that the mobile robot will obtain more expected rewards after performing the action in this state.

[0028] In a preferred embodiment, the Q-value function of the training network is further decomposed into a value function V(s; W Q ,W V ) and action advantage function A b (s,b;W Q ,W A ); where W Q 、W V and W A are the parameters for training the neural network, and and is the parameter of the target neural network; N Ω,min is the minimum experience replay buffer size, R t is the reward at the current time step t; the parameters of the target network are not updated in every training step, but are updated at a slower rate, that is, every N c After the training step.

[0029] In a preferred embodiment, the reward function is designed as follows:

[0030] R t =αf safety +βf efficiency (3)

[0031] Among them, f safety Used to evaluate the robot's performance in executing action b t The state after s t+1 The safety level of efficiency It is used to evaluate the action b t The coefficients α and β are weight factors used to adjust the importance of these two components in the reward calculation;

[0032] Specifically, f safety The reward is determined by ensuring the safety of the mobile robot based on the distance from the obstacle. It is designed as a piecewise constant function with the following form:

[0033]

[0034] Where i = 1, 2, ..., N represents the index of the HRI mode, and N is the number of specific HRI modes; 1,1,i , Υ 1,2,i , Υ 1,3,i and Y 1,4,irepresents the constant of the mobile robot with HRI modality i at different obstacle distances, while Γ1 and Γ2 are the thresholds related to the distance between the mobile robot and the obstacle; D safe is the relative safe distance between the mobile machine and the obstacle, and satisfies Γ2>Γ1>D safe relationship; D obstacle is the actual distance between the mobile robot and its nearest obstacle; similarly, f efficiency is designed as a piecewise constant function that combines the reward with the inherent properties possessed by each HRI modality:

[0035] f efficiency =Λ 1,i +Λ 2,i +Λ 3,i +Λ 4,i +Λ 5,i (5)

[0036] Among them, Λ 1,i , Λ 2,i and Λ 3,i denote the rewards associated with the delay time, human participation level, and modality recognition accuracy of the i-th modality, respectively;

[0037]

[0038] represents the ambient light intensity reward of the i-th HRI modality, and

[0039]

[0040] represents the environmental noise level reward of the i-th HRI mode; where Υ 2,1,i , Υ 2,2,i and Y 2,3,i It is expressed as the constant value for different HRI modes under different light intensity conditions; L1 and L2 are thresholds related to the ambient light intensity; 3,1,i , Υ 3,2,i and Y 3,3,i represents the constant value for different HRI modes in environments with different noise levels; P1 and P2 are thresholds related to the environmental noise level; L env and V o Represent the current environment's light intensity and ambient noise level respectively.

[0041] In a preferred embodiment, the NSBC method is used to fuse multi-modal HRI control instructions:

[0042]

[0043] Among them, u d (t n) is the fused control command, u2(t n ) has a higher priority, λ1 and λ2 are u1(t n ) and u2(t n ), is the null space of mode 2;

[0044] The Jacobian matrix is ​​modified to a time-varying form as follows:

[0045]

[0046] Among them, J i (t n )[j,j] represents the time t n When the Jacobian matrix is ​​the j-th row and j-th column element, and u i (t n )[j] represents the time t n Time control command vector u i (t n )’s jth element; Formula (7) means that for the i-th mode, if the corresponding control commands of some dimensions are not received, all-zero rows and all-zero columns will appear in its Jacobian matrix; this will affect the i-th mode at t n In particular, if there are no all-zero rows and columns in the Jacobian matrix, the null space of the mode is empty.

[0047] In a preferred embodiment, in a multimodal HRI control instruction fusion algorithm based on NSBC, the fusion process of multimodal control instructions in HRI is demonstrated by taking two modal control instructions as an example; two or more HRI modal control instructions are fused into the final control instruction according to the priority of the HRI modalities; the control instruction obtained by fusion is further fused with any new HRI modal control instruction according to the priority relationship, thereby generating new control instructions for the robot.

[0048] Compared with existing technologies, this invention offers the following advantages: Focusing on HRI systems, this invention proposes an innovative multimodal HRI method. By learning the optimal modal priority strategy offline in a simulation environment and implementing multimodal fusion based on the NSBC theory, this method enables the robot to autonomously select the modal priority that best suits its motion control. This effectively addresses the challenges of multimodal HRI, including difficulty in handling modal conflicts, low security, slow response, poor real-time performance, and lack of interpretability. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1This is a system block diagram of a multimodal fusion control method for an HRI system according to a preferred embodiment of the present invention;

[0050] Figure 2 A diagram showing the configuration steps for accessing iFLYTEK speech recognition service according to a preferred embodiment of the present invention;

[0051] Figure 3 A diagram showing a human-robot communication framework according to a preferred embodiment of the present invention;

[0052] Figure 4 Schematic representation of a modal priority action instruction according to a preferred embodiment of the present invention;

[0053] Figure 5 This is a pseudo code diagram of the DQN-based modality priority algorithm according to a preferred embodiment of the present invention;

[0054] Figure 6 A simulation environment diagram of a preferred embodiment of the present invention;

[0055] Figure 7 Graph showing experimental data results of various HRI modes in a simulation environment according to a preferred embodiment of the present invention;

[0056] Figure 8 The motion trajectory diagram of the mobile robot using three HRI modalities (language, gesture and keyboard) according to the preferred embodiment of the present invention;

[0057] Figure 9 A trajectory diagram of a preferred embodiment of the present invention using voice and gesture modalities in an environment with dim light and high noise levels;

[0058] Figure 10 This is a schematic diagram of the multi-modal control instruction fusion principle based on NSBC in a preferred embodiment of the present invention;

[0059] Figure 11 This is a pseudo code diagram of a multi-modal HRI control instruction fusion algorithm based on NSBC according to a preferred embodiment of the present invention;

[0060] Figure 12 This is a diagram of an experimental scene of a preferred embodiment of the present invention;

[0061] Figure 13 This is a theoretical architecture diagram of a system experimental platform according to a preferred embodiment of the present invention;

[0062] Figure 14 Graphs showing representative experimental traces of different HRI methods according to a preferred embodiment of the present invention;

[0063] Figure 15 This is a trajectory diagram for experimental comparison between the multimodal HRI fusion method and the weighted method in a preferred embodiment of the present invention;

[0064] Figure 16 This is a trajectory diagram of experimental results of the multimodal HRI fusion method under different environmental conditions according to a preferred embodiment of the present invention;

[0065] Figure 17 A modal priority graph for learning under four different environmental conditions over the longitudinal distance traveled by the mobile robot according to a preferred embodiment of the present invention;

[0066] Figure 18 This is a graph showing the average results of experimental data of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0067] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0068] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0069] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form, and it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0070] For a detailed description of the present invention, refer to Figure 1-18 ,The following is a detailed description of a multi-modal fusion control method for HRI system, and its specific fusion block diagram is as follows Figure 1 As shown in the figure, it can be roughly divided into three parts: the implementation of multimodal HRI basic technology, modal priority strategy learning based on reinforcement learning, and multimodal fusion based on zero-space behavior control.

[0071] Step 1: Implementation of multimodal HRI basic technology:

[0072] Step 11: Implementation of Speech Recognition

[0073] The speech recognition module uses iFlytek's speech recognition model, which collects real-time speech through the computer's microphone and then uses the Streaming Multi-Level Truncation Attention (SMLTA) speech recognition model to perform speech recognition. iFlytek has significantly improved its speech recognition accuracy by optimizing the language model. This improvement has effectively improved the efficiency of speech HRI. The system implementation process can be referred to Figure 2The standardized process shown here begins by configuring the API Key and Secret Key as security authentication credentials, then obtaining an Access Token through authorization, and finally establishing a development interface connection with the basic speech recognition model. This interface supports the secondary development of various speech functions, including in-depth customization of text-to-speech, speech recognition, and voice wake-up applications.

[0074] Step 12: Implementation of Gesture Recognition

[0075] The gesture recognition module collects gesture images through the computer's camera and then recognizes them through the gesture recognition model of the Baidu Smart Cloud Platform. It uses API calls to connect to the human body analysis service of the Baidu Smart Cloud Platform and focuses on integrating the gesture recognition function. For specific implementation plans, please refer to Figure 2 The standardized process shown in the figure: First, configure the API Key and Secret Key as authentication credentials, then obtain an Access Token through authentication, and finally establish a development interface connection with the gesture recognition model. This gesture recognition model can recognize a total of 24 gestures. The gesture recognition API supports the following 24 gestures: fist, OK, prayer, bow, goodbye, single-hand heart, thumbs up, diss, I love you, palm up, double heart (3 types), numbers (9 types), rock, and raised middle finger.

[0076] Step 13: Communication

[0077] In the system framework of the fusion control method, the communication between humans and mobile robots mainly adopts TCP. Figure 3 A communication framework between humans and mobile robots. For communication between the Robot Operating System (ROS) and its modules, including radar and binocular cameras, a set of custom ROS topic messages ( / scan / LaserScan and / camera / rgb / image_raw / Image) is created. Subscribing to these topics allows users to obtain radar and camera image information for further processing.

[0078] Step 2: Reinforcement Learning Design and Offline Training

[0079] Step 21: Action Collection and Training Process

[0080] In the context of multimodal HRI, the state space X of each HRI mode is M And the control instruction u corresponding to the user i Defined as:

[0081]

[0082] Where q is the dimension of each modal state space, N is the number of HRI modes, and u i is the discrete time t n function, and t n =nT represents the time after n samples, where T is the sampling time.

[0083] Meanwhile, in this study, the dimension of the HRI modal state space is q = 4, and we have:

[0084] u i =[x i ,y i ,ω i ,α i ] (2)

[0085] Among them, x i 、y i 、ω i and α i represent the longitudinal velocity, lateral velocity, angular velocity and absolute velocity of the i-th HRI modal control command respectively.

[0086] In a given environment Ξ, the state set S = {s t} represents various environmental conditions, where each state s t =[V o ,L env ,D obstacle ]∈R 3 Contains the noise level perceived by the robot (V o ), ambient light intensity (L env ) and the distance to the nearest obstacle (D obstacle ). Action set B = {b t} contains the priority of the modal (such as Figure 4 As shown in Figure 3, the mobile robot receives control commands from humans through these HRI modalities.

[0087] The training process of DQN is as follows:

[0088] 1) State and action selection: At time t, the robot is in state s t When, according to a decay factor γ ε The regulated ε-greedy policy selects an action b t The policy states that the robot The probability of randomly selecting an action is The action with the highest Q value is chosen with probability.

[0089] Here N E Indicates the number of training rounds completed so far.

[0090] 2) State transfer and reward: Once the robot performs action b t , the robot will receive an action execution reward R t and transfer to the new state s t+1 .

[0091] 3) Experience storage: experience tuple (s t ,b t ,R t ,s t+1 ) are then stored in an experience replay buffer called Ω. This buffer acts as a memory of past experiences, enabling them to be reused during training, thereby stabilizing the learning process and improving data efficiency.

[0092] 4) Offline training: Training proceeds to the next time step t+1, and the process continues iteratively.

[0093] During offline training, experiences are randomly sampled from Ω to update the DQN, which helps break the correlations in the observation sequence, leading to more robust learning.

[0094] 5) Iteration: This process continues until the training is complete, ensuring that the robot learns an optimal strategy for selecting actions based on the state of the environment to maximize its cumulative reward.

[0095] 6) Deterministic strategy: The goal of DQN is to obtain a deterministic strategy, expressed as π(b t ∣s t ), which optimizes the total reward accumulated by the robot. After offline training is completed, the algorithm generates a deterministic policy π(b t ∣s t ). It should be emphasized that for a specific state s t ,π(b t ∣s t ) will produce a certain action b t , rather than a probability distribution over all possible actions.

[0096] The specific implementation process pseudo code algorithm of HRI modality priority algorithm is as follows Figure 5 As shown in Algorithm 1, a training network updates its parameters to reflect the changes in the HRI modality priority strategy and is used to select the next action. At the same time, a target network provides a relatively stable target Q value, which is used to calculate the loss function during training and helps stabilize training. Q(s,b;W Q ,W V ,W A )and The Q-value functions are the Q-value functions of the training network and the target network, respectively. The Q-value function provides a quantitative metric for evaluating the quality of a specific action performed in a specific state by accumulating future rewards. A higher Q-value indicates a higher expected reward for the mobile robot after performing that action in that state. The Q-value function helps the system select the optimal HRI modality priority in different states to achieve better interaction results.

[0097] The Q-value function of the training network is further decomposed into the value function V(s; W Q ,W V ) and action advantage function A b (s,b;W Q ,W A The advantage of decomposing the Q function into a value function and an advantage function is that it can achieve generalization by learning actions without changing the underlying reinforcement learning algorithm framework. Q 、W V and W A are the parameters for training the neural network, and and are the parameters of the target neural network. N Ω,min is the minimum experience replay buffer size, R t is the reward at the current time step t. The parameters of the target network are not updated in every training step, but are updated at a slower rate, usually every N c After the training step.

[0098] Step 22: Reward Design

[0099] Specifically, the reward function is designed as follows:

[0100] R t =αf safety +βf efficiency (3)

[0101] Among them, f safety Used to evaluate the robot's performance in executing action b t The state after s t+1 The safety level of efficiency It is used to evaluate the action b t The efficiency of the reward is determined by the reduction in human involvement, the latency of the HRI modality, and the accuracy (which may be affected by noise levels and light intensity). The coefficients α and β are weighting factors used to adjust the importance of these two components in the reward calculation. These two coefficients can be adjusted accordingly based on the specific requirements and constraints of the application scenario.

[0102] Specifically, f safetyThe reward is determined by ensuring the safety of the mobile robot based on the distance from the obstacle. It is designed as a piecewise constant function with the following form:

[0103]

[0104] Where i=1,2,…,N represents the index of HRI mode, and N is the total number of specific HRI modes.

[0105] Υ 1,1,i , Υ 1,2,i , Υ 1,3,i and Y 1,4,i represents the constant of the mobile robot under different obstacle distances for the i-th HRI modal, while Γ1 and Γ2 are the thresholds related to the distance between the mobile robot and the obstacle. safe is the relative safe distance between the mobile machine and the obstacle, and satisfies Γ2>Γ1>D safe relationship. D obstacle is the actual distance between the mobile robot and its nearest obstacle. Similarly, f efficiency is designed as a piecewise constant function that combines the reward with the intrinsic properties possessed by each HRI modality:

[0106] f efficiency =Λ 1,i +Λ 2,i +Λ 3,i +Λ 4,i +Λ 5,i (5)

[0107] Among them, Λ 1,i , Λ 2,i and Λ 3,i denote the rewards associated with the delay time, human participation level, and modality recognition accuracy of the i-th modality, respectively.

[0108]

[0109] represents the ambient light intensity reward of the i-th HRI modality, and

[0110]

[0111] represents the environmental noise level reward of the i-th HRI mode. 2,1,i , Υ 2,2,i and Y 2,3,i It is represented as a constant value for different HRI modes under different light intensity conditions. L1 and L2 are thresholds related to the ambient light intensity. 3,1,i , Υ 3,2,i and Y 3,3,iIndicates the constant value for different HRI modes in environments with different noise levels. P1 and P2 are thresholds related to the environmental noise level. env and V o Represent the current environment's light intensity and ambient noise level respectively.

[0112] Step 23: Simulation environment experiment and offline training

[0113] The offline training of Algorithm 1 is Figure 6 The simulation was conducted in the environment shown, where a mobile robot navigates while avoiding obstacles on its way to a destination. Rectangular area 1 represents a wide open space, while rectangular areas 2 and 3 represent complex environments with a higher obstacle density and narrower paths.

[0114] Figure 7 The figure shows the average task completion time, recognition accuracy, and user engagement using three interaction modalities, each with its own unique advantages and disadvantages. For example, the keyboard modality demonstrated extreme flexibility and 100% command input accuracy, with the shortest average latency of 1.08 milliseconds and the shortest task completion time. However, it imposed the greatest cognitive burden on humans. Conversely, the voice modality required the least human engagement and cognitive burden, although its performance metrics were suboptimal. While the gesture modality performed slightly better than the voice modality, it lagged behind the keyboard modality in terms of accuracy. Furthermore, it can be observed that the recognition accuracy rates for the voice modality and gesture modality were 91.05% and 92.36%, respectively. This suggests that collisions with obstacles are possible when controlling a mobile robot using these two modalities.

[0115] The motion trajectory of the mobile robot using three HRI modalities (language, gesture and keyboard) is shown in Figure 8 As shown in the figure. It can be clearly seen that the inherent high latency of the voice modality (1.108 seconds on average) leads to poor control performance, which may pose a potential danger to the robot when it approaches an obstacle. In addition, when controlling the mobile robot to the target point, whether due to the delay or because the recognition accuracy is not 100%, the robot does not stop at the target point immediately, but moves a distance after accurately reaching the destination. In the trajectory represented by the gesture modality, there are two obvious defects in controlling the mobile robot: one is the misrecognition of gestures, which leads to deviation from the expected control path; the other is the delay in gesture recognition (0.385 seconds on average). Therefore, the robot did not reach the target point through the expected shortest path, but took a longer route and continued to travel a distance beyond the target point.

[0116] In addition, the same simulation experiments were conducted in a dim light and high noise level environment, and the results are as follows: Figure 9As shown in the figure, it can be observed that due to the influence of environmental factors (sound and light), the robot control performance of voice and gesture interaction modes is poor, and robots controlled by these two modes collide with obstacles. Based on simulation experiments, it can be clearly seen that the use of different HRI modes has different effects on the time required to control the mobile robot to complete the task, the recognition accuracy of control commands, and the workload of the human. In addition, the environment also has an impact on the HRI mode, such as the impact of light on gesture interaction and the impact of noise on voice interaction. Based on simulation experiments, different rewards are designed for each HRI mode during the offline training process of Algorithm 1.

[0117] Step 3: NSBC-based control instruction fusion

[0118] Based on the modal priority learned by Algorithm 1, the NSBC method is then used to fuse the multi-modal control instructions. In order to introduce the present invention in detail, the following are three cases of HRI modes. The principle of multi-modal control instruction fusion based on the NSBC method is as follows: Figure 10 In this figure, I represents the identity matrix, J i represents the Jacobian matrix of the i-th HRI mode, and represents the null space of the i-th HRI mode. The priority of control instructions from different modes is u2>u1>u3. The final control instruction u d It is obtained by first projecting u3 into the null space of u1, and then projecting the obtained control instructions into the null space of u2 for fusion.

[0119] The traditional NSBC is extended to support asynchronous input of HRI modal control instructions. This means that within the same sampling time T, the input control instructions can be a single HRI modal or multiple HRI modalities. In contrast, the traditional synchronous input of HRI modal control instructions means that the user can input the control instructions at time t. n Multiple HRI modal control instructions are input simultaneously, and then the system fuses these inputs using the NSBC method according to the following formula:

[0120]

[0121] Among them, u d (t n ) is the fused control command, u2(t n ) has a higher priority, λ1 and λ2 are u1(t n ) and u2(t n ), is the null space of mode 2.

[0122] In order to adapt to the asynchronous input of the HRI modal control command, the Jacobian matrix is ​​modified to a time-varying form as follows:

[0123]

[0124] Among them, J i (t n )[j,j] represents the time t n When the Jacobian matrix is ​​the j-th row and j-th column element, and u i (t n )[j] represents the time t n Time control command vector u i (t n ). Formula (4-7) means that for the i-th mode, if the corresponding control commands of some dimensions are not received, all-zero rows and all-zero columns will appear in its Jacobian matrix. This will affect the i-th mode at t n In particular, if there are no all-zero rows and columns in the Jacobian matrix, the null space of the mode is empty.

[0125] The pseudo code of the multi-modal HRI control instruction fusion algorithm based on NSBC is as follows: Figure 11 As shown in Figure 2 , this algorithm demonstrates the fusion process of multimodal control instructions in HRI using two modal control instructions as an example. It is also worth noting that two or more HRI modal control instructions can be fused into the final control instruction based on the priority of the HRI modalities. Furthermore, the fused control instruction can be further fused with any new HRI modal control instruction based on the priority relationship, thereby generating new control instructions for the robot.

[0126] Step 4: Experimental comparison and analysis

[0127] In the experimental case given, the present invention designed a Figure 12 The experimental environment shown and Figure 13The theoretical architecture of the experimental setup is presented. The experiment requires the user to control the mobile robot from a starting point to a target point using different modalities (including keyboard, voice, and gestures). The specific experimental setup is the same as in Chapter 3 and will not be further described here. It should also be noted that this experiment differs substantially from the simulation experiments, demonstrating the generalization capability of the proposed method. Due to the limited size of the experimental site, obstacle distances were categorized into three levels: near, medium, and far, corresponding to specific numerical ranges of [0m, 0.4m], [0.4m, 0.7m], and greater than 0.7m, respectively. Furthermore, two environmental factors, light intensity and noise intensity, were standardized. Specifically, light intensity was categorized into three levels: normal, dim, and dark, corresponding to numerical ranges of [1, 0.6], [0.3, 0.6], and [0, 0.3], respectively; noise intensity was also categorized into three levels: low, medium, and high, corresponding to numerical ranges of [0, 0.4], [0.4, 0.7], and [0.7, 1], respectively. The distances to obstacles around the mobile robot were determined using the vehicle's own radar and camera. Noise detection is done by the host computer's microphone. The ambient brightness level is measured using images captured by the vehicle's onboard camera.

[0128] The proposed multimodal HRI fusion method was compared with multiple single-modal HRIs and a weighted fusion-based multimodal HRI method. In the weighted fusion-based multimodal HRI method, the weights of the three HRI modalities were: keyboard -0.4, speech -0.3, and gesture -0.3. Specifically, the performance of different HRI methods was tested under four different environmental conditions: normal lighting and low noise intensity, dim lighting and high noise intensity, and a combination of dim lighting and high noise intensity.

[0129] In the experiment, the representative experimental results are as follows Figure 14 As shown in Figure 2. The driving trajectory of the robot using multimodal HRI can be divided into three segments: In an open environment (the first segment), voice or gesture modalities are used to reduce the burden on the user. In a complex obstacle environment (the second segment), the keyboard modality is used to better control the vehicle. In the last segment with a moderate number of obstacles, the keyboard modality and voice modality are combined to achieve effective obstacle avoidance and task completion. Figure 4-9 As can be seen in Figure 2, the keyboard modality and multimodal HRI successfully avoided obstacles and reached the target point. The voice modality encountered two collisions when interacting with the circular and rectangular obstacles, while the gesture modality encountered one collision when interacting with these obstacles.

[0130] The proposed multimodal HRI intelligent fusion method is further compared with the weighted fusion method. Figure 15The figure shows the trajectory of a vehicle in an experiment. The vehicle, controlled by the weighted fusion method, collided with a circular obstacle and its trajectory was not smooth as it approached the target point. This is because the weighted fusion method uses fixed weights for all modes, resulting in poor adaptability in complex environments.

[0131] To verify the robustness of the proposed method, we conducted experiments under different environmental conditions. Figure 16 The vehicle trajectories from a single experiment under four environmental conditions (normal environment, low light, high noise, and low light and high noise) are shown. It can be seen that the proposed multimodal HRI method successfully avoids obstacles and reaches the target point under different environmental conditions by intelligently adjusting the priority order of the modalities.

[0132] from Figure 17 (a) It can be seen that under normal environmental conditions, the proposed method selects Mode 2 (Voice > Keyboard > Gesture) within the first 800 mm distance because there are almost no obstacles in the first section of the experiment, so the voice modality is sufficient for safe and accurate motion control. From a distance of approximately 1000 mm to 1500 mm, the proposed method selects Mode 0 and Mode 1, both of which give the keyboard modality the highest priority. This is because the keyboard modality is most suitable for accurate obstacle avoidance. In contrast, from Figure 17 (d) It can be seen that under conditions of loud noise and low light, the priority order is mode 0 first, then mode 1, that is, the keyboard mode is set to the highest priority to ensure that the vehicle can effectively avoid obstacles and complete the task when the voice and gesture modes are invalid.

[0133] Figure 18 Results for three metrics obtained using different methods under different environmental conditions are presented. The results show that both keyboard and multimodal interaction achieve relatively short task completion times, with multimodal interaction being slightly faster under normal conditions. This is because controlling an autonomous vehicle using a keyboard in an environment with many obstacles is complex, requiring multiple key presses for acceleration, deceleration, and directional control. The proposed multimodal method allows for acceleration using voice and gesture modalities, while the keyboard modality is used for directional control and obstacle avoidance, resulting in smoother and more efficient control. As expected, under the other three environmental conditions, the multimodal HRI method exhibits slightly longer task completion times relative to normal conditions, but the increase is minimal. Another observation is that the gesture modality has shorter latency than the voice modality, which also contributes to the reduction in average task completion time. Unsurprisingly, the weighted fusion method, due to its fixed weights for all modalities, requires significantly longer task completion times than the proposed multimodal HRI method.

[0134] Furthermore, under different environmental conditions, neither the keyboard modality nor the multimodal HRI method experienced collision accidents, while the voice and gesture modalities did collide due to their lower recognition accuracy. In particular, the voice modality experienced an average of 1.91 collisions per trial, nearly two collisions per trial. This suggests that using either the voice or gesture modality alone is undesirable. However, the proposed multimodal HRI method alleviates this problem, providing safe motion control while reducing human involvement compared to keyboard interaction alone. In summary, the effectiveness and superiority of the developed multimodal HRI fusion method compared to single-modality and weighted modality fusion methods are well demonstrated.

[0135] The above description is only a preferred embodiment of the present invention. The present invention is not limited to the above embodiment. Any local changes, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal fusion control method for a human-computer interaction system, characterized in that: The following steps are involved: Step 1: Implementation of multimodal HRI basic technology; Step 2: Modality priority strategy learning based on reinforcement learning; By combining the Deep Q-Network (DQN) algorithm in reinforcement learning with a reinforcement learning modality priority supervision system, the HRI system autonomously learns modality priority strategies based on safety and efficiency criteria. In a simulated environment, the HRI system trains the DQN by comprehensively considering obstacle distance, modal delay, recognition accuracy, light intensity, and noise level indicators. Step 3: Use the NSBC method to fuse multi-modal HRI control instructions: project the low-priority modal instructions to the null space of the high-priority modal instructions.

2. A multimodal fusion control method for a human-computer interaction system according to claim 1, characterized in that: The step 1 includes the implementation of speech recognition, the implementation of gesture recognition and communication; speech recognition is performed using the speech recognition module, gesture recognition is performed through the gesture recognition module, and the keyboard of the host computer is used to combine into a multimodal HRI, and the corresponding recognition results are generated as control instructions.

3. The multimodal fusion control method for a human-computer interaction system according to claim 1, characterized in that: In step 2, different HRI modal control instructions are used as the action set of the DQN algorithm, and a suitable mobile robot state set, such as the position of the mobile robot, is selected as the state set of the DQN algorithm. A reasonable reward function is designed to train the DQN by comprehensively considering obstacle distance, modal delay, recognition accuracy, light intensity, and noise level indicators.

4. The multimodal fusion control method for a human-computer interaction system according to claim 3, characterized in that: In the context of multimodal HRI, the state space X of each HRI mode is M And the control instruction u corresponding to the user i Defined as: Where q is the dimension of each mode state space, N is the number of HRI modes, and t n =nT represents the time after n sampling times, where T is the sampling time; The dimension of the HRI modal state space is q = 4, and we have: you i =[x i ,y i ,oh i ,a i ] (2) Among them, x i 、y i 、ω i and α i They represent the longitudinal velocity, lateral velocity, angular velocity and absolute velocity of the i-th HRI modal control command respectively; In a given environment Ξ, the state set S = {s t } represents various environmental conditions, where each state s t =[V o ,L env ,D obstacle ]∈R 3 Contains the noise level V perceived by the robot o , ambient light intensity L env and the distance D to the nearest obstacle obstacle ; Action set B = {b t }Contains the priority of the modality, and the mobile robot receives control commands from humans through the HRI modality.

5. The multimodal fusion control method for a human-computer interaction system according to claim 4, characterized in that: The training process of the deep Q network algorithm DQN is as follows: 1) State and action selection: At time t, the robot is in state s t When, according to a decay factor γ ε The regulated ε-greedy policy selects an action b t ; The ε-greedy strategy stipulates that the robot The probability of randomly selecting an action is The probability of selecting the action with the highest Q value is E Indicates the number of training rounds completed so far; 2) State transfer and reward: Once the robot performs action b t , the robot will receive an action execution reward R t and transfer to the new state s t+1 ; 3) Experience storage: experience tuple (s t ,b t ,R t ,s t+1 ) are then stored in an experience replay buffer called Ω; this buffer acts as a memory of past experiences that are reused during training, stabilizing the learning process and improving data efficiency; 4) Offline training: Training proceeds to the next time step t+1, and the process continues iteratively. During offline training, experiences are randomly drawn from Ω to update the deep Q network algorithm DQN. 5) Iteration: This process continues until training is complete, ensuring that the robot learns an optimal strategy for selecting actions based on the state of the environment to maximize its cumulative reward. 6) Deterministic strategy: The goal of the deep Q network algorithm DQN is to obtain a deterministic strategy, expressed as π(b t ∣s t ), the deterministic strategy optimizes the total reward accumulated by the robot; after offline training is completed, the deep Q network algorithm DQN will generate a deterministic strategy π(b t ∣s t ); It should be emphasized that for a specific state s t ,π(b t ∣s t ) will produce a certain action b t , rather than a probability distribution over all possible actions.

6. The multimodal fusion control method for a human-computer interaction system according to claim 5, characterized in that: The HRI modality priority algorithm includes training the network Q(s,b;W Q ,W V ,W A ) and a target network, the training network is used to update its parameters to reflect the changes in the HRI modality priority strategy and to select the next action; the target network provides a stable target Q value, which is used to calculate the loss function during training; the Q value function of the target network is The Q-value function provides the robot with a quantitative indicator of the quality of performing a specific action in a specific state by accumulating future rewards; a higher Q-value indicates that the mobile robot will obtain more expected rewards after performing the action in this state.

7. The multimodal fusion control method for a human-computer interaction system according to claim 6, characterized in that: The Q-value function of the training network is further decomposed into the value function V(s; W Q ,W V ) and action advantage function A b (s,b;W Q ,W A ); Among them, W Q 、W V and W A are the parameters for training the neural network, and and is the parameter of the target neural network; N Ω,min is the minimum experience replay buffer size, R t is the reward at the current time step t; the parameters of the target network are not updated in every training step, but are updated at a slower rate, that is, every N c After the training step.

8. The multimodal fusion control method for a human-computer interaction system according to claim 7, characterized in that: The reward function is designed as follows: R t =αf safety +βf efficiency (3) Among them, f safety Used to evaluate the robot's performance in executing action b t The state after s t+1 The safety level of efficiency It is used to evaluate the action b t The coefficients α and β are weight factors used to adjust the importance of these two components in the reward calculation; Specifically, f safety The reward is determined by ensuring the safety of the mobile robot based on the distance from the obstacle. It is designed as a piecewise constant function with the following form: Where i = 1, 2, ..., N represents the index of the HRI mode, and N is the number of specific HRI modes; 1,1,i , Υ 1,2,i , Υ 1,3,i and Y 1,4,i represents the constant of the mobile robot with HRI modality i at different obstacle distances, while Γ1 and Γ2 are the thresholds related to the distance between the mobile robot and the obstacle; D safe is the relative safe distance between the mobile machine and the obstacle, and satisfies Γ2>Γ1>D safe relationship; D obstacle is the actual distance between the mobile robot and its nearest obstacle; similarly, f efficiency is designed as a piecewise constant function that combines the reward with the inherent properties possessed by each HRI modality: f efficiency =L 1,i +L 2,i +L 3,i +L 4,i +L 5,i (5) Among them, Λ 1,i , Λ 2,i and Λ 3,i denote the rewards associated with the delay time, human participation level, and modality recognition accuracy of the i-th modality, respectively; represents the ambient light intensity reward of the i-th HRI modality, and represents the environmental noise level reward of the i-th HRI mode; where Υ 2,1,i , Υ 2,2,i and Y 2,3,i It is expressed as the constant value for different HRI modes under different light intensity conditions; L1 and L2 are thresholds related to the ambient light intensity; 3,1,i , Υ 3,2,i and Y 3,3,i represents the constant value for different HRI modes in environments with different noise levels; P1 and P2 are thresholds related to the environmental noise level; L env and V o Represent the current environment's light intensity and ambient noise level respectively.

9. The multimodal fusion control method for a human-computer interaction system according to claim 1, characterized in that: Adopting NSBC method to fuse multi-modal HRI control instructions: Among them, u d (t n ) is the fused control command, u2(t n ) has a higher priority, λ1 and λ2 are u1(t n ) and u2(t n ), is the null space of mode 2; The Jacobian matrix is ​​modified to a time-varying form as follows: Among them, J i (t n )[j,j] represents the time t n When the Jacobian matrix is ​​the j-th row and j-th column element, and u i (t n )[j] represents the time t n Time control command vector u i (t n )’s jth element; Formula (7) means that for the i-th mode, if the corresponding control commands of some dimensions are not received, all-zero rows and all-zero columns will appear in its Jacobian matrix; this will affect the i-th mode at t n In particular, if there are no all-zero rows and columns in the Jacobian matrix, the null space of the mode is empty.

10. The multimodal fusion control method for a human-computer interaction system according to claim 1, characterized in that: In the NSBC-based multimodal HRI control instruction fusion algorithm, the fusion process of multimodal control instructions in HRI is demonstrated by taking two modal control instructions as an example; two or more HRI modal control instructions are fused into the final control instruction according to the priority of the HRI modalities; the control instruction obtained by fusion is further fused with any new HRI modal control instruction according to the priority relationship, thereby generating new control instructions for the robot.

Images