Robot voice interaction system based on multi-modal sensing fusion and control method thereof

Abstract

The invention discloses a robot voice interaction system based on multi-mode sensing fusion and a control method thereof, and belongs to the technical field of intelligent robots, and the control method comprises the steps: S1, the initial state of a robot is a dormant state; s2, awakening the robot by collecting awakening words in real time; s3, performing dynamic compensation according to a sound source and IMU data fusion algorithm, performing beam forming optimization correction on the sound source, and performing pickup to obtain voice data; s4, detecting and obtaining rule / intention voice; s5, recognizing the rule / intention voice, and performing NLU semantic understanding through the cloud ASR or the offline ASR to obtain a voice instruction; s6, the voice instruction is fused with visual data of the detection sensor group for dynamic arbitration, and the voice instruction or the obstacle avoidance instruction is executed; and S7, after the instruction in the step S6 is executed, entering a dormant state so as to achieve the purpose that a voice interaction system is formed by fusing the microphone array, the IMU sensor and edge calculation, and the intelligent robot autonomously decides in a complex environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent robots, and more particularly, relates to a robot voice interaction system based on multi-modal sensing fusion and a control method thereof. Background Art

[0002] An intelligent robot is a mechanical device integrating a variety of advanced technologies, with the capabilities of perception, decision-making, and execution. It can imitate or execute human behaviors and tasks, and is a product that highly combines physical and mental labor, aiming to build an artificial machine that can "think".

[0003] A voice robot belongs to a type of intelligent robot. It is an automated system that uses artificial intelligence technologies, especially technologies such as automatic speech recognition (ASR), natural language processing (NLP), and text-to-speech (TTS), to automatically make phone calls and conduct intelligent conversations. It can provide telephone services through voice interaction and has functions such as automatic dialing, speech recognition, natural language understanding and generation, information push, data recording and analysis.

[0004] The pain points of existing voice robots are as follows:

[0005] (1) Low recognition rate due to far-field noise interference

[0006] In far-field speech recognition scenarios, due to factors such as environmental noise, reverberation, and multiple people speaking, the recognition rate of the speech recognition system decreases.

[0007] (2) Insufficient semantic understanding ability in offline scenarios

[0008] When the voice robot is in an offline state, its semantic understanding ability is poor, resulting in a poor user experience.

[0009] (3) Low multi-sensor collaboration efficiency

[0010] The main reason for the low multi-sensor collaboration efficiency of intelligent robots is the lack of an effective collaboration mechanism. Multi-agent systems (MAS) are widely used in fields such as cooperation, navigation, and scheduling due to their distributed decision-making and self-organization capabilities. However, due to the lack of an effective collaboration mechanism, these systems often face problems such as low collaboration efficiency and low task completion rate.

[0011] Based on the above, there is an urgent need to develop a robot voice interaction system based on multi-modal sensing fusion to overcome the above defects. Summary of the Invention

[0012] In view of this, in order to solve the above problems existing in the prior art, the purpose of the present invention is to provide a robot voice interaction system based on multi-modal sensing fusion and its control method, so as to achieve the purpose of forming a voice interaction system by fusing a microphone array, an IMU sensor and edge computing, and enabling an intelligent robot to make autonomous decisions in complex environments.

[0013] The technical solution adopted by the present invention is as follows: A robot voice interaction system based on multi-modal sensing fusion, the system includes:

[0014] A main control chip;

[0015] A TPU processor communicatively connected to the main control chip, the TPU processor is connected to a sensor component through a digital signal processor;

[0016] A motion module communicatively connected to the main control chip, the motion module executes motion actions according to the motion instructions of the main control chip;

[0017] A speaker and a voice recognition server respectively communicatively connected to the main control chip;

[0018] Wherein, the TPU processor is a coprocessor of the main control chip.

[0019] Further, the sensor component includes a microphone array, an IMU sensor and an ultrasonic radar, the microphone array, the IMU sensor and the ultrasonic radar are respectively connected to the digital signal processor, and the ultrasonic radar is communicatively connected to the main control chip.

[0020] Further, the motion module includes:

[0021] A motion controller, the motion controller is communicatively connected to a driving motor, and the motion controller controls the movement of the driving motor through a PWM signal;

[0022] A dynamic arbiter, the main control chip is communicatively connected to the motion controller through the dynamic arbiter.

[0023] Further, the voice recognition server includes: an ASR local server and an ASR cloud server.

[0024] The present invention also discloses a control method for robot voice interaction based on multi-modal sensing fusion, the control method includes:

[0025] S1: The initial state of the robot is the sleep state;

[0026] S2: Wake up the robot by real-time collecting wake-up words;

[0027] S3: Obtain the sound source data and IMU data, perform dynamic compensation according to the sound source and IMU data fusion algorithm, and perform beamforming optimization and correction on the sound source, and then pick up the sound to obtain the voice data;

[0028] S4: Perform VAD endpoint detection on the voice data to obtain regular / intentional voice;

[0029] S5: Recognize the regular / intentional voice. If the current network is good, perform NLU semantic understanding through cloud ASR to obtain the voice command; if the current network has a delay, perform NLU semantic understanding through offline ASR to obtain the voice command;

[0030] S6: Dynamically arbitrate by fusing the voice command with the visual data of the detection sensor group. If no obstacle is detected, execute the voice command; if an obstacle is detected, issue an obstacle avoidance command;

[0031] S7: After the command in S6 is executed, enter the sleep state.

[0032] Further, in S3, the sound source and IMU data fusion algorithm includes:

[0033] S301: Synchronize the time after preprocessing the sound source data collected by the microphone array and the IMU data collected by the IMU sensor;

[0034] S302: Coarsely locate the sound source according to the sound source data;

[0035] S303: Dynamically compensate the sound source positioning according to the IMU data;

[0036] S304; Optimize and correct the beamforming of the microphone array after calculating the dynamic compensation.

[0037] Further, the calculation method of the dynamic compensation includes:

[0038] S3031: Detect that the yaw angle change angle is α through the IMU sensor;

[0039] S3032: Calculate the dynamic compensation amount Δ through the IMU dynamic compensation algorithm, and Δ = α + β, where β is the mechanical offset calibration amount;

[0040] S3033: Adjust the center of the main lobe of the beamforming of the microphone array to the direction where Δ is located.

[0041] Further, in S5, if the network delay > 100ms, it will automatically switch to offline ASR.

[0042] Further, in S6, the method of the dynamic arbitration includes:

[0043] S601: Identify the voice command and visual data respectively;

[0044] S602: If the visual data is identified as an obstacle avoidance signal, the priority of the obstacle avoidance signal is higher than that of the voice command, and the arbitration executes the obstacle avoidance command; if the visual data does not identify an obstacle avoidance signal, the priority of the voice command is higher than that of the obstacle avoidance signal, and the arbitration executes the voice command;

[0045] S603: Feed the obstacle avoidance command or voice command back to the motion controller, and convert it into an execution command through the motion controller.

[0046] Further, the detection sensor group includes an ultrasonic sensor and a radar, and the ultrasonic sensor and the radar are used to detect visual data on the path.

[0047] The beneficial effects of the present invention are as follows:

[0048] 1. By adopting the robot voice interaction system based on multi-modal sensing fusion provided by the present invention, the system combines a main control chip and a Coral TPU coprocessor to form a hardware heterogeneous computing architecture, integrates a microphone array and a sensor module, and hands over the motion instruction control and decision execution to the main control chip, which can make autonomous decisions on the movement of the robot in a complex environment and enhance the collaborative ability in a multi-sensor scenario.

[0049] 2. By adopting the robot voice interaction system based on multi-modal sensing fusion provided by the present invention, the main control chip of the system is communicatively connected to an offline or cloud server, and its semantic understanding ability in the offline scenario is significantly improved.

[0050] 3. By adopting the robot voice interaction control method based on multi-modal sensing fusion provided by the present invention, after the IMU data, the robot is dynamically compensated for sound source localization, thereby enhancing directional sound pickup and having a higher speech recognition rate. At the same time, after semantic understanding, it can perform logical analysis on voice commands and obstacle avoidance signals through an arbitration algorithm, and combine with a multi-modal data priority decision mechanism to finally achieve optimal control of the movement path of the robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is the overall logic block diagram of the robot voice interaction system based on multi-modal sensing fusion provided by the present invention;

[0052] Figure 2 is the logic flowchart of the robot voice interaction control method based on multi-modal sensing fusion provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0053] Embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar modules or modules with the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and should not be construed as a limitation of the present application. On the contrary, the embodiments of the present application include all changes, modifications, and equivalents that fall within the spirit and scope of the appended claims.

[0054] Embodiment 1

[0055] As Figure 1 shown, this embodiment provides a robot voice interaction system based on multi-modal sensing fusion. The system includes: a main control chip and a TPU processor. The TPU processor is communicatively connected to the main control chip, and the TPU processor is a coprocessor of the main control chip. A hardware heterogeneous computing architecture is formed by combining the main control chip and the Coral TPU coprocessor. The TPU processor is connected to a sensor component through a digital signal processor. Among them, the digital signal processor is a DSP processor. The DSP processor optimizes FFT through a dedicated instruction set to improve the processing efficiency of digital signals, and the role of the DSP processor is to perform real-time noise reduction. Specifically, the main control chip adopts NVIDIA Jetson Nano / Xavier, etc., which supports parallel voice processing. The built-in voice module of the main control chip adopts ESP32-LyraT, etc., which has a low cost and is suitable for simple instructions. The coprocessor adopts Google Coral TPU, which can perform wake word detection by accelerating neural network inference.

[0056] The above-mentioned sensor component includes a microphone array, an IMU sensor, and an ultrasonic radar. The microphone array, IMU sensor, and ultrasonic radar are respectively connected to the digital signal processor, and the ultrasonic radar is communicatively connected to the main control chip. The IMU data collected by the IMU sensor is directly transmitted using DMA. Specifically, the microphone array is a 6-microphone circular array (extended from ReSpeaker 4-Mic Array). The microphone array supports noise reduction and sound source localization, and adopts a low-cost ReSpeaker chip or a high-performance Synaptics AudioSmart chip. The microphone array cooperates with the DSP processor to perform real-time noise reduction. At the same time, this robot is pre-allocated with a double-buffered audio memory pool to store audio data in real time. The IMU sensor adopts a 6-axis sensor of BMI088 with a 100Hz update rate, which is used for the robot to perform orientation compensation, thereby enhancing directional sound pickup. The ultrasonic / radar sensor is used for the input of visual data, thereby assisting in obstacle avoidance in the later stage.

[0057] There is also a motion module communicatively connected to the main control chip, and the motion module executes motion actions according to the motion instructions of the main control chip; specifically, the motion module includes:

[0058] A motion controller, which is communicatively connected between the motion controller and the drive motor, and the motion controller controls the motion of the drive motor through PWM signals. The drive motor executes the displacement motion of the robot through a corresponding motion chassis;

[0059] A dynamic arbiter, which is communicatively connected between the main control chip and the motion controller through the dynamic arbiter. During operation, the dynamic arbiter can execute the conflict handling logic of voice commands and obstacle avoidance signals, and arbitrate to obtain the best control command according to the logic that the voice command priority is lower than the ultrasonic emergency obstacle avoidance signal.

[0060] The above main control chip is communicatively connected to a speaker and a voice recognition server respectively. Among them, the voice recognition server includes: an ASR local server and an ASR cloud server. The speaker can adopt a high-fidelity speaker, such as a JBL embedded speaker, to achieve the purpose of external output.

[0061] The working principle of the voice interaction system provided in this embodiment is as follows:

[0062] After voice input through the microphone array for voice recognition, when detecting regular voices such as "move forward or backward" or intent voices such as "avoid the table or stay away from the stool", it is connected to the main control chip through ROS low-level communication, or serial port or UDP. The main control chip issues instructions to control the movement of the robot. During operation, the brain also needs to fuse visual data such as ultrasonic or radar data for dynamic arbitration to avoid obstacles first.

[0063] Embodiment 2

[0064] As Figure 2 shown, the present invention also discloses a control method for robot voice interaction based on multi-modal sensing fusion, and the control method includes:

[0065] S1: The initial state of the robot after power-on is the sleep state, and the sleep state can also be understood as the standby state;

[0066] S2: Wake up the robot by collecting wake-up words in real time. Specifically, use a lightweight model Porcupine or Snowboy to wake up the voice, and custom wake-up words are supported;

[0067] S3: Obtain sound source data and IMU data, perform dynamic compensation according to the sound source and IMU data fusion algorithm, and perform beamforming optimization and correction on the sound source to pick up voice data. The sound source and IMU data fusion algorithm includes:

[0068] S301: Synchronize the sound source data collected by the microphone array and the IMU data collected by the IMU sensor in time after preprocessing. In this embodiment, the PTP protocol is used to align the timestamps, and the error is <1ms. The preprocessing includes: calibrating the collected IMU data by eliminating high-frequency jitter through Kalman filtering; performing audio data framing on the collected sound source data and storing the audio data of the most recent 200ms in a circular buffer.

[0069] S302: Coarse localize the sound source based on the sound source data. The method for coarse localizing the sound source is: Generalized Cross-Correlation - Phase Transform (GCC-PHAT);

[0070] S303: Dynamically compensate the sound source localization according to the IMU data. The calculation method of this dynamic compensation includes:

[0071] S3031: Detect that the change angle of the yaw angle is α through the IMU sensor;

[0072] S3032: Calculate the dynamic compensation amount Δ through the IMU dynamic compensation algorithm, and Δ = α + β, where β is the mechanical offset calibration amount; among them, the calculation process of the dynamic compensation amount Δ is:

[0073] (1) Coordinate system mapping:

[0074] Microphone position vector: v_mic = [x1, y1, z1;...; x6, y6, z6]

[0075] Sound source direction vector: v_source = [cosθ, sinθ, 0] (assuming the horizontal plane)

[0076] (2) Rotation transformation compensation:

[0077] Δ = arctan2((R(t) @ v_mic_mean × v_source)[2], (R(t) @

[0078] v_mic_mean) · v_source) + β

[0079] β: Mechanical offset calibration amount, this parameter is obtained through laser calibration;

[0080] Among them, the rotation matrix R(t) is calculated through the IMU data, also known as: Euler angle to rotation matrix, R(t) = R_yaw(y) × R_pitch(p) × R_roll(r); this rotation matrix R(t) accelerates matrix operations through the Coral TPU processor.

[0081] S3033: Adjust the center of the main lobe of the beamforming of the microphone array to the direction where Δ is located, and when the angular velocity variance var(ω) of the robot > the threshold, increase the beam width σ;

[0082] S304: After dynamic compensation calculation, optimize and correct the beamforming of the microphone array. Among them, the optimization method of beamforming is through spatial domain weighting optimization and spatial domain filtering optimization, specifically as follows:

[0083] (1) Frequency domain weighting optimization method (STFT layer):

[0084] Apply the direction weight W(θ’) to the microphone signal X_k(f) to enhance the energy in the target direction. The calculation formula of the direction weight W(θ’) is:

[0085] W(θ’) = exp(-(θ’ - μ) 2 / (2σ 2 )) * (1 - α · var(ω))

[0086] Among them, θ’ refers to the spatial direction variable, which is used to calculate the weights at different angles and can be a static scanning angle or the real-time direction of a dynamic target; u is the spatial pointing center of the beam, σ is the beam width, var(ω) is the angular velocity variance, the degree of dynamic change of the frequency domain parameters; α is the change angle of the yaw angle, the stability compensation intensity, and the larger the value, the stronger the suppression of dynamic interference;

[0087] Realize frequency domain focusing through the Gaussian main lobe, and dynamically suppress the instability caused by motion or environmental changes through the variance compensation mechanism. It is a frequency domain weighting optimization method that takes into account both static performance and dynamic robustness.

[0088] (2) Spatial domain filtering optimization method (MVDR layer):

[0089] Construct the steering vector ɑ(θ’), and the formula for calculating the MVDR (Minimum Variance Distortionless Response) beamformer is:

[0090]

[0091] Among them, R xx : The autocorrelation matrix of the input signal (including noise and interference); R xx -1 : The inverse of the autocorrelation matrix, which is used to suppress noise and interference;

[0092] ɑ(θ’): The steering vector, which represents the array response in the target direction (the angle is θ’);

[0093] ɑ H (θ’) is: conjugate transpose;

[0094] If Rxx is the identity matrix (with only white noise), the weights degenerate into the normalized steering vector, and the beam naturally points to the target direction; in the presence of interference, R xx -1 automatically adjusts the weights to suppress interference.

[0095] S4: Perform VAD endpoint detection on the speech data to obtain regular / intentional speech, and for speech synthesis, use Google WaveNet or Azure Neural TTS in the cloud; use eSpeak or Mozilla TTS neural network locally.

[0096] S5: Recognize the regular / intentional speech. If the current network is good, perform NLU semantic understanding through cloud ASR to obtain the speech command; if the current network has latency, perform NLU semantic understanding through offline ASR to obtain the speech command; among them, cloud ASR uses Google Speech-to-Text or Alibaba Cloud ASR; offline ASR uses the Vosk offline open-source model or NVIDIA Riva, which has the characteristic of low latency. In this step, the load balancing strategy of offline / cloud hybrid ASR is: if the network latency > 100ms, then automatically switch to offline ASR and preferentially select the Vosk offline open-source model. In practical applications, the recognition of intentional speech uses the Rasa open-source framework or the Google Dialogflow platform.

[0097] S6: Fuse the speech command with the visual data of the detection sensor group for dynamic arbitration. If no obstacle is detected, execute the speech command; if an obstacle is detected, perform an obstacle avoidance command; among them, the detection sensor group includes ultrasonic sensors and radars, and the ultrasonic sensors and radars are used to detect the visual data on the path. Among them, the method of the dynamic arbitration includes:

[0098] S601: Recognize the speech command and the visual data respectively;

[0099] S602: If the visual data is recognized as an obstacle avoidance signal, the priority of the obstacle avoidance signal is greater than that of the speech command, and the arbitration executes the obstacle avoidance command; if the visual data does not recognize the obstacle avoidance signal, the priority of the speech command is greater than that of the obstacle avoidance signal, and the arbitration executes the speech command;

[0100] S603: Feed back the obstacle avoidance command or the speech command to the motion controller, and convert it into an execution command through the motion controller.

[0101] Specifically, the priority arbitration rule:

[0102] Signal type Priority Response action Ultrasonic emergency obstacle avoidance Highest Immediately stop the execution of voice commands IMU fall detection High Trigger the security protocol and issue a voice alarm Navigation target point instruction Medium Store in the task queue Normal voice interaction Low Execute normally

[0103] S7: After the instruction in S6 is executed, enter the sleep state.

[0104] Based on the control method provided in this embodiment, its technical advantages compared with the traditional method are as follows:

[0105] Indicator Traditional method This technology Recognition accuracy during movement <60％ >85％ Response delay 50 - 100ms <20ms Power consumption 3.5W 1.2W (TPU + DSP collaboration)

[0106] By jointly calibrating the IMU data and the sound source data of the microphone array, sub-second sound source tracking accuracy in a dynamic environment is achieved.

[0107] It should be noted that any process or method description in the flowchart or described in other ways herein can be understood as representing a module, segment, or part of executable instructions including one or more steps for implementing a specific logical function or process, and the scope of the preferred implementation mode of this application includes additional implementations, where the functions can be executed in a manner essentially the same as or in the reverse order according to the involved functions, rather than in the order shown or discussed, which should be understood by those skilled in the technical field to which the embodiments of this application belong.

[0108] It should be understood that each part of this application can be implemented by hardware, software, firmware, or a combination thereof. In the above implementation mode, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another implementation mode, any one or a combination of the following well-known technologies in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits with appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0109] Those of ordinary skill in the technical field of this application can understand that all or part of the steps carried by the method of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0110] In addition, in each embodiment of this application, each functional unit can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. When the above integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0111] The above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disc, etc.

[0112] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A robot voice interaction system based on multimodal sensing fusion, characterized in that, The system includes: A main control chip; A TPU processor communicatively connected to the main control chip, and the TPU processor is connected to a sensor component through a digital signal processor; A motion module communicatively connected to the main control chip, and the motion module executes motion actions according to the motion instructions of the main control chip; A speaker and a voice recognition server respectively communicatively connected to the main control chip; Wherein, the TPU processor is a coprocessor of the main control chip.

2. The robot voice interaction system based on multimodal sensing fusion according to claim 1, wherein The sensor component includes a microphone array, an IMU sensor, and an ultrasonic radar. The microphone array, the IMU sensor, and the ultrasonic radar are respectively connected to the digital signal processor, and there is a communication connection between the ultrasonic radar and the main control chip.

3. The robot voice interaction system based on multimodal sensing fusion according to claim 1, wherein The motion module includes: A motion controller communicatively connected to a driving motor, and the motion controller controls the driving motor to move through a PWM signal; A dynamic arbiter, and the main control chip is communicatively connected to the motion controller through the dynamic arbiter.

4. The robot voice interaction system based on multi-modal sensing fusion according to claim 1, characterized in that, The voice recognition server includes: an ASR local server and an ASR cloud server.

5. A control method for robot voice interaction based on multi-modal sensing fusion, characterized in that, The control method includes: S1: The initial state of the robot is a sleep state; S2: Wake up the robot by collecting wake-up words in real time; S3: Obtain sound source data and IMU data, perform dynamic compensation according to the sound source and IMU data fusion algorithm, and perform beamforming optimization and correction on the sound source to pick up voice data; S4: Perform VAD endpoint detection on the voice data to obtain regular / intentional voice; S5: Recognize the regular / intentional voice. If the current network is good, perform NLU semantic understanding through cloud ASR to obtain voice instructions; if the current network has a delay, perform NLU semantic understanding through offline ASR to obtain voice instructions; S6: Dynamically arbitrate by fusing the voice instructions with the visual data of the detection sensor group. If no obstacle is detected, execute the voice instructions; if an obstacle is detected, issue an obstacle avoidance instruction; S7: After the instruction in S6 is executed, enter the sleep state.

6. The control method for robot voice interaction based on multi-modal sensing fusion according to claim 5, characterized in that, In S3, the sound source and IMU data fusion algorithm includes: S301: Synchronize the sound source data collected by the microphone array and the IMU data collected by the IMU sensor in time after preprocessing; S302: Coarsely locate the sound source according to the sound source data; S303: Perform dynamic compensation on the sound source location according to the IMU data; S304: Optimize and correct the beamforming of the microphone array according to the calculation after dynamic compensation.

7. The control method for robot voice interaction based on multi-modal sensing fusion according to claim 6, wherein the calculation method of the dynamic compensation includes: S3031: Detect that the yaw angle change angle is α through the IMU sensor; S3032: Calculate the dynamic compensation amount Δ through the IMU dynamic compensation algorithm, and Δ = α + β, where β is the mechanical offset calibration amount; S3033: Adjust the center of the main lobe of the beamforming of the microphone array to the direction where Δ is located.

8. The control method for robot voice interaction based on multi-modal sensing fusion according to claim 5, in S5, if the network delay > 100ms, then automatically switch to offline ASR.

9. The control method for robot voice interaction based on multimodal sensing fusion according to claim 5, in S6, the method of dynamic arbitration includes: S601: Identify voice commands and visual data respectively; S602: If the visual data is identified as an obstacle avoidance signal, the priority of the obstacle avoidance signal is greater than that of the voice command, and the arbitration executes the obstacle avoidance command; if the visual data does not identify an obstacle avoidance signal, the priority of the voice command is greater than that of the obstacle avoidance signal, and the arbitration executes the voice command; S603: Feed back the obstacle avoidance command or the voice command to the motion controller, and convert it into an execution command through the motion controller.

10. The control method for robot voice interaction based on multimodal sensing fusion according to claim 5, the detection sensor group includes ultrasonic sensors and radars, and the ultrasonic sensors and radars are used to detect visual data on the path.

Images