ROS-based robot integrated multi-mode fusion intelligent interaction system and method

Abstract

The invention discloses an ROS-based robot integrated multi-mode fusion intelligent interaction system and method, and the system comprises a monitoring layer which comprises a computer and a man-machine interaction module, subscribes information collected by a sensor in a decision layer through an ROS, carries out the real-time voice recognition and gesture recognition, and displays a recognition result, a control instruction corresponding to an identification result is issued to a decision-making layer carried by the robot through an ROS topic, or a motion control instruction is directly issued through the ROS topic and sent to an execution layer; the decision-making layer is carried at a robot end and comprises an edge embedded calculation module and a sensor, and the edge embedded calculation module receives the control instruction issued by the man-machine interaction module, carries out auxiliary decision making in combination with information collected by the sensor and sends an action instruction corresponding to the control instruction to the execution layer through an ROS topic message; according to the execution layer, the robot subscribes to the action instruction issued by the decision-making layer or the man-machine interaction module through the ROS to complete the specified motion. According to the invention, the use of more Ubuntu terminal commands in the ROS use process is avoided, and the problems of poor integration of software and hardware of a robot system, incomplete functions of a control system and the like are solved.

Description

Technical Field

[0001] The present invention relates to the field of ROS robot human-machine interaction, and in particular to a robot integrated multi-modal fusion intelligent interaction system and method based on ROS. Background Art

[0002] With the rapid development of artificial intelligence and robotics, robots are increasingly used in industries such as industry, services, and medical care. Modern robots not only need to perform complex tasks, but also need to have the ability to interact naturally with humans. Using a variety of perception and control methods such as voice and gestures, robots can interact with humans more naturally and efficiently, so as to play a greater role in human-machine collaboration and intelligent services. However, existing multimodal interaction systems often lack a unified integration platform, making it difficult to achieve efficient modular design and control. Traditional robot systems also face many challenges in achieving efficient human-machine interaction, including unfriendly interaction interfaces, single interaction methods, low system integration, high reliance on terminal commands, and complex operations. These problems limit the popularity and application of ROS robots among non-professional users. Summary of the invention

[0003] The purpose of the present invention is to provide a robot integrated multimodal fusion intelligent interaction system and method based on ROS, so as to solve the problems that the existing ROS robots only use ROS terminal commands, handles and keyboard controls, resulting in complex operation, single interaction mode, poor system integration and the inability to meet the robot multimodal intelligent interactive control needs.

[0004] The technical solution to achieve the purpose of the present invention is:

[0005] A robot integrated multi-modal fusion intelligent interaction system based on ROS, comprising:

[0006] The monitoring layer, including computers and human-computer interaction modules, subscribes to the information collected by sensors in the decision-making layer through ROS, performs real-time speech recognition and gesture recognition, displays the recognition results, and sends the control instructions corresponding to the recognition results to the decision-making layer on the robot through the ROS topic, or directly sends motion control instructions to the execution layer through the ROS topic;

[0007] The decision-making layer is installed on the robot side and includes an edge embedded computing module and sensors. The edge embedded computing module receives the control instructions issued by the human-computer interaction module, makes auxiliary decisions based on the information collected by the sensors, and sends the action instructions corresponding to the control instructions to the execution layer through ROS topic messages;

[0008] At the execution layer, the robot subscribes to the action instructions issued by the decision-making layer or the human-computer interaction module through ROS to complete the specified movement.

[0009] Furthermore, the human-computer interaction module includes a speech recognition module, a gesture recognition module, a motion control button, a robot motion state module and a front-view camera module. The speech recognition module and the gesture recognition module are respectively used for speech and gesture recognition and interactive control, the motion control button is used to send action instructions for the robot, the robot motion state module is used to display the motion state of the robot, and the front-view camera module is used to display images taken by the robot's front-view camera.

[0010] Furthermore, the motion control buttons include direction keys and gait control keys.

[0011] Furthermore, the human-computer interaction module is developed using Qt, and the speech recognition module or gesture recognition module is started, and a speech or gesture script is automatically generated in Qt's ROS. If the script has content, it is cleared first, and relevant commands for speech recognition and gesture recognition are written, and the script execution permission is granted, and finally the script is executed.

[0012] Furthermore, the speech recognition module adopts a Paraformer model, each speech recording duration is 2 seconds, and the Paraformer model is connected to the motion instruction word classification module. After the Paraformer model recognizes the corresponding instruction in the motion instruction word classification module, it sends the instruction to the ROS topic for the decision-making layer to subscribe.

[0013] Furthermore, the gesture recognition module adopts the Mediapipe framework, the Mediapipe framework is connected to the motion command word classification module, the hand tracking module of Mediapipe is used to detect and identify the position and posture of the key points of the hand, and the corresponding instructions are obtained through the motion command word classification module, and the corresponding instructions are sent to the ROS topic for the decision-making layer to subscribe.

[0014] Furthermore, the command words of the motion command word classification module include forward, backward, turn left, turn right, move left, move right, stand up, lie down, stop and start.

[0015] Furthermore, the interaction between the decision layer and the monitoring layer and the execution layer specifically includes: first, based on the fusion baseline period, the confidence of each instruction issued by the speech recognition module is obtained, and sent to the speech fusion ROS topic; the gestures recognized by the gesture recognition module in the historical baseline period and the confidence corresponding to each gesture are obtained, and sent to the gesture fusion ROS topic; then the information fusion ROS node is initialized, and the speech fusion ROS topic and the gesture fusion ROS topic are subscribed to at the same time to obtain the instructions and confidence of the speech and gestures. If the input of one of the speech or gesture is repeatedly obtained, the latter will cover the same data obtained in the previous time until the other modality is input and linear weighted fusion is performed. After linear weighted fusion, the instruction with the highest confidence is obtained, and normalized, and finally the instruction with the highest confidence is output through the ROS topic for subscription by the execution layer robot.

[0016] Furthermore, the output of the linear weighted fusion is:

[0017] η i =H i +ps i

[0018]

[0019] Among them, ps i is the confidence of the speech recognition command Ci, and defines ph i , k is the command C for gesture recognition in each frame i , k = 1, 2, ...mi, mi is the number of speech recognition instruction results of the i-th frame, and N represents the number of instructions.

[0020] A robot integrated multi-modal fusion intelligent interaction method based on ROS, comprising:

[0021] Click the human-computer interaction module to perform real-time speech recognition and gesture recognition, display the recognition results, and send the control instructions corresponding to the recognition results to the decision-making layer on the robot through the ROS topic, or directly send the motion control instructions to the execution layer through the ROS topic;

[0022] The decision layer receives the control instructions issued by the human-computer interaction module, makes auxiliary decisions based on the information collected by the sensor, and sends the action instructions corresponding to the control instructions to the execution layer through ROS topic messages;

[0023] The robot subscribes to the action instructions issued by the decision-making layer or the human-computer interaction module through ROS to complete the specified movement.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] (1) The present invention provides a control method for a monitoring layer-decision layer-execution layer interactive system based on the ROS architecture, forming a highly integrated interactive system. The layers communicate in real time through ROS topics, ensuring the accuracy and real-time nature of information transmission. The system design is open and flexible and applicable to a variety of robot application scenarios. Users can not only directly control the robot body through the upper computer monitoring layer, but also verify and adjust it through the simulation interface, thereby enhancing the applicability and expansibility of the system.

[0026] (2) The monitoring layer of the present invention uses Qt Creator to integrate modules such as speech recognition, gesture recognition, front-view camera, and various motion control buttons, which can display information such as the robot's motion status and current time. The Paraformer model is used to achieve efficient and accurate speech recognition, and the Mediapipe framework is used to achieve real-time gesture recognition. By clicking the corresponding module once, the speech and gesture recognition functions in ROS can be enabled. This not only provides a multi-modal human-computer interaction method, but also improves the user experience and ease of operation, allowing non-professionals to use the interactive system well.

[0027] (3) The present invention uses weighted fusion and normalization processing to fuse the recognition results of speech and gestures, achieving more natural and accurate interaction. The decision-making layer information fusion ensures the output of the highest confidence command, improves the accuracy of the interaction and the reliability of the system, and also enhances the robustness of the system in complex environments, promoting the development of intelligent interaction technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the overall structure of an embodiment of the monitoring layer-decision layer-execution layer interactive system control method based on the ROS architecture of the present invention.

[0029] Figure 2 It is a simplified diagram of ROS topic communication between the monitoring layer, decision layer and execution layer in the present invention.

[0030] Figure 3 This is a diagram of a ROS robot control experiment performed through an interactive system in an embodiment of the present invention.

[0031] Figure 4 It is a schematic diagram of the trajectory of the simulated robot controlled by the interactive system in the Rviz visual interface in an embodiment of the present invention.

[0032] Figure 5 It is an interface diagram of the ROS robot multimodal human-computer interaction system mounted on the monitoring layer host computer of the present invention.

[0033] Figure 6 It is a process logic flow chart of the process enabled by the human-computer interaction interface system of the present invention.

[0034] Figure 7 This is a diagram for realizing the fusion of speech recognition and gesture recognition information at the decision-making layer of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below in conjunction with the accompanying drawings.

[0036] A robot integrated multimodal fusion intelligent interaction system based on ROS, with a monitoring layer-decision-making layer-execution layer interaction design based on the ROS architecture; the overall structure is mainly composed of the monitoring layer, decision-making layer and execution layer. The monitoring layer provides a multimodal human-computer interaction system interface; the decision-making layer performs information fusion, aiming to provide a more open and flexible solution for the research and development and application of robots.

[0037] The monitoring layer is composed of an Ubuntu computer and a designed human-computer interaction software interface, which is responsible for subscribing to audio, video, robot motion status and other information collected by sensors in the decision-making layer through ROS, and realizing real-time speech recognition and gesture recognition in the human-computer interaction software interface, and sending the control instructions corresponding to the recognition results to the decision-making layer computer NVIDIA Jetson AGX Orin (edge ​​embedded computing module) carried by the robot. In the embodiment, the Rviz visualization interface can also be started through the human-computer interaction software of the present invention and the motion control of the simulation robot can be realized, which further verifies the feasibility of this method.

[0038] The decision-making layer is composed of NVIDIA JetsonAGX Orin (edge ​​embedded computing module) and sensors. It has a certain computing power and is responsible for receiving control instructions issued by the upper computer human-computer interaction software and making auxiliary decisions based on the information collected by the sensors. Finally, the action code corresponding to the control instruction is sent to the execution layer through the ROS topic message.

[0039] The execution layer is mainly composed of actuators such as the four-wheeled legged robot body or the Rviz visualization interface simulation robot. It completes the specified movement finally achieved by the actuator through the action code issued by the ROS subscription decision layer or the human-computer interaction software system. Through this design, users can realize the interactive control from the host computer monitoring layer to the actuator body.

[0040] The multimodal human-computer interaction system interface includes a startup interface and a system interface. The startup interface consists of a human-computer interaction background picture and an enter program button. After clicking the enter program button, you can jump to the human-computer interaction system interface. The system interface can display information such as the system name, voice recognition module, gesture recognition module, forward-looking camera, various motion control buttons, robot motion status, current time, etc.

[0041] Click the speech recognition module or gesture recognition module in the interface, and a shell script for speech or gesture will be automatically generated in the Qt ROS project. If the script has content, clear it first, write commands related to speech recognition and gesture recognition such as enabling the terminal and running related ROS function packages or launch files, grant execution permissions to the related shell scripts, and finally execute the script. The above operations can be completed with one click.

[0042] After clicking the voice recognition module, the voice interaction interface will be opened, and the voice interaction ROS node will be initialized at the same time, and real-time voice interaction control can be performed. The voice recognition adopts the Paraformer model, which is based on the Transformer architecture and combines the parameter-efficient design to achieve efficient and accurate voice recognition. During use, the duration of each voice recording is limited to 2 seconds, and the motion command word classification module is connected to the back end of the model. The command words include: "forward", "backward", "turn left", "turn right", "move left", "move right", "stand", "lie down", "stop", "start" and other 10 words. After the model recognizes the corresponding command through voice, it sends the command to the ROS topic for the decision-making layer to subscribe.

[0043] After clicking the gesture recognition module, the gesture interaction interface will be opened, real-time gesture interaction control will be started, and the gesture interaction ROS node will be initialized. Gesture recognition adopts the Mediapipe framework (real-time multimedia processing framework). Mediapipe has cross-platform and lightweight deployment characteristics. This patent focuses on hand detection and uses the hand key point detection module Hand Tracking (hand tracking module) provided by Mediapipe to detect and identify the position and posture of 21 key points of the hand. With voice recognition, 10 gestures are set, and the recognized gestures are mapped with the above 10 command words to complete gesture recognition and classification, and the recognized commands are sent to the ROS topic for the decision-making layer to subscribe.

[0044] The motion control buttons, such as direction keys, gait control keys, etc., send various motion instructions to ROS topics / chatter by opening the ROS node initialized by the human-computer interaction software interface to complete the basic motion control of the robot.

[0045] The decision-making layer information fusion combines the advantages of both speech recognition and gesture recognition to achieve more natural and accurate human-computer interaction, overcoming the limitations of single modality recognition technology. A longer speech recognition cycle is selected as the fusion baseline cycle. Speech recognition outputs the confidence of each instruction and sends it to the speech fusion ROS topic. Gesture recognition collects all gestures recognized in the historical baseline cycle and the confidence corresponding to each gesture, and sends it to the gesture fusion ROS topic. Then initialize the information fusion ROS node and subscribe to the above two topics at the same time to obtain the instructions and confidence of speech and gesture. If the input of one of speech or gesture is obtained repeatedly, the latter will cover the same data obtained before, until the other modality is input, weighted fusion is performed to ensure the latest and most accurate fusion results. The instruction with the highest confidence is obtained through weighted fusion and normalized. Finally, the instruction with the highest confidence is output through the ROS topic for subscription by the execution layer robot.

[0046] A robot integrated multi-modal fusion intelligent interaction method based on ROS, comprising:

[0047] Click the human-computer interaction module to perform real-time speech recognition and gesture recognition, display the recognition results, and send the control instructions corresponding to the recognition results to the decision-making layer on the robot through the ROS topic, or directly send the motion control instructions to the execution layer through the ROS topic;

[0048] The decision layer receives the control instructions issued by the human-computer interaction module, makes auxiliary decisions based on the information collected by the sensor, and sends the action instructions corresponding to the control instructions to the execution layer through ROS topic messages;

[0049] The robot subscribes to the action instructions issued by the decision-making layer or the human-computer interaction module through ROS to complete the specified movement.

[0050] The present invention uses Qt and ROS to design an interaction method for the monitoring layer, decision-making layer, and execution layer, develops a multimodal human-computer interaction interface, and completes command fusion at the decision-making layer, thereby realizing high-precision, low-latency robot control, significantly improving the integration and interactive performance of the robot system, and solving the shortcomings of traditional robot systems in terms of software and hardware integration and control functions.

[0051] Example

[0052] like Figure 1 As shown, this embodiment provides a robot integrated multimodal fusion intelligent interactive system based on ROS, with Ubuntu20.04 as the operating system and Qt Creator as the interactive interface development platform. The entire system is based on the distributed processing framework of the robot control system (ROS) to carry out the design of the monitoring layer-decision-making layer-execution layer interactive system control method, and realizes high-precision and low-latency control from the interactive interface to the robot. Figure 1 It contains the components and communication logic between each layer. The following will further combine Figure 2 , that is, the ROS topic communication diagram between the monitoring layer, decision layer and execution layer is described as follows:

[0053] The monitoring layer is composed of an Ubuntu computer and a designed human-computer interaction software interface. In this embodiment, after starting the human-computer interaction software interface of the monitoring layer host computer, the speech and gesture recognition modules are turned on. The speech, gesture recognition and basic key control modules in the software will respectively initialize their respective nodes: / speech, / hand and / key, and send the speech, gesture recognition instructions and key control instructions as messages to the topic / chatter. If the decision-making layer information fusion is enabled, the system will initialize the / data_fusion node, subscribe to the output of speech recognition and gesture recognition, and perform confidence weighted fusion and normalization steps, and finally output the fusion result to the topic / chatter for subscription by the decision-making layer AGX Orin and Rviz robot simulation system.

[0054] The decision layer is composed of NVIDIA JetsonAGX Orin and sensors. It has a certain computing power and is responsible for receiving control instructions issued by the upper computer human-computer interaction software and making auxiliary decisions based on the information collected by the sensors. In the decision layer, AGXOrin initializes the / talker_listener node and receives motion instructions by subscribing to the / chatter topic. After receiving the instructions, it will check the current robot status through the / robot_basic_state topic. If the robot status meets the status requirements before issuing the instruction, it will construct a command message corresponding to the motion instruction, that is, the encapsulated action code, and send the action code message to the topic / control / command.

[0055] The execution layer is mainly composed of actuators such as the quad-legged robot body or the Rviz visualization interface simulation robot. It completes the specified movement finally achieved by the actuator by subscribing to the action code issued by the decision layer or the human-computer interaction software system through ROS. The robot chassis in the execution layer obtains the control parameters of each joint by subscribing to the topic / control / command and sends them to the robot chassis drive motor and other modules, thereby controlling the quad-legged robot to perform corresponding actions and realize the robot's motion control. The experimental process is as follows Figure 3As shown; if the simulation system is enabled, the Rviz robot simulation system in the execution layer receives motion instructions through the / chatter topic, completes the mapping of motion instructions to the actual defined kinematic parameters through the transfer node / command_to_rviz, and sends out simulation robot motion messages through the topic / cmd_vel for the simulation robot node / quadruped_simulator to subscribe to, and finally displays the simulation results in the Rviz interface by subscribing to the topic / wb_state. The simulation results are shown in the figure below. Figure 4 shown.

[0056] The use of Qt combined with ROS to develop a multimodal human-computer interaction system interface requires the installation of the corresponding version of the ROS plug-in in Qt Creator. Only after the plug-in is installed can the ROS workspace be created in Qt creator, the ROS node be initialized, etc. for subsequent communication construction, system design, etc. The present invention uses a more advanced package of catkin_make compilation, and configures the required QT design functions and the dependent libraries and resource files required for various functions one by one in the CMakeLists.txt file under the created function package, including the UI function design library, the opencv library, etc.

[0057] like Figure 5 As shown, the multimodal human-computer interaction system interface in the present invention includes a startup interface and a system interface. The startup interface consists of a human-computer interaction background picture and an enter program button. After the configuration of the master and slave IP addresses is set up and ROS is enabled, click the enter program button to jump to the human-computer interaction system interface. The system interface can display information such as the system name, voice recognition module, gesture recognition module, front-view camera, various motion control buttons, robot motion status, current time, etc. At this time, the module is enabled, and communication will be carried out through the ROS topic. The ROS robot can be controlled through the module in the interface. If the enable simulation control button is clicked, the Rviz simulation interface will be enabled. At this time, the simulated robot can be controlled through human-computer interaction, and the current motion status information of the robot can be displayed through the status feedback module. The system interface terminal enables the process logic as shown in Figure 6 As shown, it includes: configuring IP address, establishing ROS communication between monitoring layer, decision layer and execution layer, opening system application, judging whether ROS communication is established, if it is established, entering human-computer interaction interface, clicking button of human-computer interaction interface, communicating with decision layer and execution layer on ROS topic, completing robot action, and closing system program when control action is completed, otherwise continue to complete instruction action. The following will further introduce the implementation logic of each module.

[0058] Click the speech recognition module or gesture recognition module in the interface, and according to the pre-written code, the shell script of speech or gesture will be automatically regenerated in the ROS project of Qt. If the script has content, it will be cleared first, and the relevant commands of speech recognition and gesture recognition such as enabling the terminal and running the relevant ROS function package or launch file will be written, and the relevant shell script execution permission will be granted, and finally the script will be executed. The above operations can be completed with one click. This not only takes care of the user experience but also ensures the availability of the system after transplantation.

[0059] After clicking the voice recognition module, the voice interaction interface will be opened, and the voice interaction ROS node will be initialized at the same time, and real-time voice interaction control can be performed. The voice recognition adopts the Paraformer model. During use, the present invention limits the duration of each voice recording to 2 seconds, and connects the motion instruction word classification module to the back end of the model. The instruction words include: "forward", "backward", "turn left", "turn right", "move left", "move right", "stand up", "lie down", "stop", "start" and other 10 words. After the model recognizes the corresponding instruction through voice, it will send the instruction to the ROS topic / chatter for the decision-making layer to subscribe.

[0060] After clicking the gesture recognition module, the gesture interaction interface will be opened, real-time gesture interaction control will be started, and the gesture interaction ROS node will be initialized. Gesture recognition adopts the Mediapipe framework and uses the hand key point detection module Hand Tracking provided by Mediapipe to detect and identify the position and posture of 21 key points of the hand. Same as speech recognition, 10 gestures are set, and the recognized gestures are mapped to the above 10 command words to complete gesture recognition and classification, and the recognized commands are sent to the ROS topic / chatter for the decision-making layer to subscribe.

[0061] The human-computer interaction system for ROS intelligent robot control is used to solve the complex and tedious problems involving many commands and Ubuntu terminal operations during the use of ROS intelligent robots.

[0062] The decision-level information fusion achieves a more natural and accurate human-computer interaction by combining the advantages of both speech recognition and gesture recognition, thus overcoming the limitations of single-modality recognition technology.

[0063] The decision-making layer information fusion is implemented as follows: speech recognition outputs the confidence of each command and sends it to the speech fusion ROS topic / speech_part; gesture recognition collects all recognized gestures and the confidence corresponding to each gesture within the historical baseline period, and sends them to the gesture fusion ROS topic / hand_part; then initializes the information fusion ROS node / data_fusion, and subscribes to the above two topics at the same time to obtain the commands and confidence of speech and gestures; in the fusion system, if one of the speech or gesture inputs is received repeatedly, the later data will overwrite the previously obtained data of the same type until the other modality input is weighted fusion to ensure the latest and most accurate fusion results. Information fusion is implemented as follows Figure 7 shown.

[0064] Furthermore, the specific formula for information fusion is implemented as follows:

[0065] Choose a longer speech recognition period T s As the fusion baseline cycle, it is expressed as shown in formula 1:

[0066] Ts = Ts_end - Ts_start (1)

[0067] Where T_start is the time when speech recognition starts, T_end is the time when speech recognition ends, and the period Ts in the present invention is fixed to 2 seconds. s In the definition of ps i is the confidence of the speech recognition command Ci.

[0068] Definition of ph i , j (j = 1, 2, ...mi) is the command C for gesture recognition in each frame i The confidence value is output by the model in period T s The set of internal gesture recognition results can be defined as W. In this paper, 10 instructions are designed, so N∈(1, 2, ..., 10), as shown in Formula 2:

[0069]

[0070] In the gesture recognition result set W, it is recognized as command C i The specific gravity is defined as rh i , as shown in Formula 3:

[0071]

[0072] Defines the command to be recognized as C i The average confidence of all gestures is as follows: i Formula 4 shows:

[0073]

[0074] In the current cycle T s In the command, it is recognized as command C i The total confidence of all gestures H i It is defined as shown in Formula 5:

[0075]

[0076] After the above processing, in each cycle T s The total confidence H of gesture recognition is obtained i It has higher applicability and rationality.

[0077] Finally, perform gesture recognition H in the same cycle i With speech recognition ps i The linear weighted fusion of i , and then normalized to get Finally choose the largest Value command C i , the command is still issued through the / chatter topic. The specific process is shown in formula 6-8.

[0078] η i =H i +ps i (6)

[0079]

[0080] Those skilled in the art can understand and implement all or part of the steps in the above-mentioned embodiment method. The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above-mentioned embodiment. Any equivalent modification or change made by those skilled in the art based on the content disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. A robot integrated multi-modal fusion intelligent interaction system based on ROS, characterized in that: include: The monitoring layer, including computers and human-computer interaction modules, subscribes to the information collected by sensors in the decision-making layer through ROS, performs real-time speech recognition and gesture recognition, displays the recognition results, and sends the control instructions corresponding to the recognition results to the decision-making layer on the robot through the ROS topic, or directly sends motion control instructions to the execution layer through the ROS topic; The decision-making layer is installed on the robot side and includes an edge embedded computing module and sensors. The edge embedded computing module receives the control instructions issued by the human-computer interaction module, makes auxiliary decisions based on the information collected by the sensors, and sends the action instructions corresponding to the control instructions to the execution layer through ROS topic messages; At the execution layer, the robot subscribes to the action instructions issued by the decision-making layer or the human-computer interaction module through ROS to complete the specified movement.

2. According to the ROS-based robot integrated multi-modal fusion intelligent interaction system according to claim 1, it is characterized in that: The human-computer interaction module includes a speech recognition module, a gesture recognition module, a motion control button, a robot motion state module and a front-view camera module. The speech recognition module and the gesture recognition module are respectively used for speech and gesture recognition and interactive control. The motion control button is used to send action instructions for the robot. The robot motion state module is used to display the motion state of the robot. The front-view camera module is used to display images taken by the robot's front-view camera.

3. The ROS-based robot integrated multi-modal fusion intelligent interaction system according to claim 2, characterized in that: The motion control buttons include direction keys and gait control keys.

4. The ROS-based robot integrated multi-modal fusion intelligent interaction system according to claim 2, characterized in that: The human-computer interaction module is developed using Qt, and the speech recognition module or gesture recognition module is started. A speech or gesture script is automatically generated in Qt's ROS. If the script has content, it is cleared first, and the relevant commands for speech recognition and gesture recognition are written, and the script execution permission is granted, and finally the script is executed.

5. The ROS-based robot integrated multi-modal fusion intelligent interaction system according to claim 4, characterized in that: The speech recognition module adopts the Paraformer model. Each speech recording time is 2 seconds. The Paraformer model is connected to the motion instruction word classification module. After the Paraformer model recognizes the corresponding instruction in the motion instruction word classification module, it sends the instruction to the ROS topic for the decision-making layer to subscribe.

6. The ROS-based robot integrated multi-modal fusion intelligent interaction system according to claim 5, characterized in that: The gesture recognition module adopts the Mediapipe framework, which is connected to the motion instruction word classification module. The Mediapipe hand tracking module is used to detect and identify the position and posture of the key points of the hand, and the corresponding instructions are obtained through the motion instruction word classification module. The corresponding instructions are sent to the ROS topic for the decision layer to subscribe.

7. The ROS-based robot integrated multi-modal fusion intelligent interaction system according to claim 5, characterized in that: The instruction words of the motion instruction word classification module include forward, backward, turn left, turn right, move left, move right, stand up, lie down, stop and start.

8. The ROS-based robot integrated multi-modal fusion intelligent interaction system according to claim 2, characterized in that: The interaction between the decision layer, the monitoring layer and the execution layer specifically includes: first, based on the fusion baseline period, the confidence of each instruction issued by the speech recognition module is obtained, and sent to the speech fusion ROS topic; the gestures recognized by the gesture recognition module in the historical baseline period and the confidence corresponding to each gesture are obtained, and sent to the gesture fusion ROS topic; then the information fusion ROS node is initialized, and the speech fusion ROS topic and the gesture fusion ROS topic are subscribed to at the same time to obtain the instructions and confidence of the speech and gestures. If the input of one of the speech or gesture is repeatedly obtained, the latter will cover the same data obtained in the previous time until the other modality is input and linear weighted fusion is performed. After linear weighted fusion, the instruction with the highest confidence is obtained, and normalized, and finally the instruction with the highest confidence is output through the ROS topic for subscription by the execution layer robot.

9. The ROS-based robot integrated multi-modal fusion intelligent interaction system according to claim 8, characterized in that: The output of the linear weighted fusion is: η i =H i +ps i Among them, ps i is the confidence of the speech recognition command Ci, and defines ph i , k is the command C for gesture recognition in each frame i , k = 1, 2, ...mi, mi is the number of speech recognition instruction results of the i-th frame, and N represents the number of instructions.

10. A ROS-based robot integrated multi-modal fusion intelligent interaction method based on the system of any one of claims 1-9, characterized in that: include: Click the human-computer interaction module to perform real-time speech recognition and gesture recognition, display the recognition results, and send the control instructions corresponding to the recognition results to the decision-making layer on the robot through the ROS topic, or directly send the motion control instructions to the execution layer through the ROS topic; The decision layer receives the control instructions issued by the human-computer interaction module, makes auxiliary decisions based on the information collected by the sensor, and sends the action instructions corresponding to the control instructions to the execution layer through ROS topic messages; The robot subscribes to the action instructions issued by the decision-making layer or the human-computer interaction module through ROS to complete the specified movement.

Images