Robot voice interaction and autonomous behavior decision-making method and system based on multi-modal perception

Abstract

The invention discloses a robot voice interaction and autonomous behavior decision-making method and system based on multi-modal perception, and belongs to the field of autonomous control and intelligent interaction of service robots. The problems that an existing service robot technology cannot meet the natural language interaction requirement in a complex scene, and precise understanding and autonomous decision making of task intentions cannot be achieved are solved. The invention provides a natural language analysis mechanism based on weighted confidence fusion, which performs confidence evaluation and weighted fusion on multi-modal information such as speech recognition, semantic analysis, visual perception, positioning perception and the like to realize intelligent disassembly of composite statements, multi-round instructions and parameter-containing instructions. Accurate understanding and autonomous decision making of task intentions are achieved, and therefore the robot is accurately driven to act. The method is mainly applied to the service robot.

Description

Technical Field

[0001] This invention belongs to the field of autonomous control and intelligent interaction of service robots. Background Technology

[0002] With the improvement of artificial intelligence processing power and the popularization of sensor technology, the application of service robots in smart education, exhibition explanation, and commercial services is growing rapidly. However, in actual deployment, existing service robots still face several technical bottlenecks in key technologies such as voice interaction, semantic parsing, and autonomous behavior decision-making. First, traditional speech recognition systems rely heavily on limited instruction templates and cannot perform deep analysis of the semantic structure in speech, resulting in a significant drop in recognition accuracy in noisy environments, free-flowing spoken expression, or multi-turn interaction scenarios. In addition, existing semantic understanding methods are usually based on keyword matching or rule trees, lacking the ability to comprehensively reason about semantic elements such as action sequence, operational logic, and parameter relationships. Second, service robots generally lack end-to-end behavior mapping capabilities from natural language to executed actions, making it difficult to generate structured instructions that can directly drive navigation controllers, motion planning algorithms, and task execution modules based on users' natural expressions. This necessitates manually setting up a large number of state machines and rule programs to maintain task logic, reducing the system's scalability and robustness.

[0003] In terms of interactive experience, existing robots typically lack a unified scheduling mechanism between voice interaction and autonomous navigation. The voice module, navigation module, and task module operate independently, lacking shared context and state management, making it impossible for them to perform real-time path updates, task switching, and behavior planning based on dialogue content. For example, when a user requests, "Take me to a certain exhibition area and explain the relevant content," traditional systems cannot handle this in a unified manner and still require multiple manual triggering commands. Furthermore, robots underutilize multimodal perception, failing to integrate and understand information such as voice, vision, and spatial positioning, leading to unstable behavior and disjointed interactions in complex and dynamic environments. In summary, existing technologies cannot meet the needs of natural language interaction in complex scenarios, nor can they achieve accurate understanding of task intent and autonomous decision-making. There is an urgent need for an integrated intelligent system and method that combines multimodal perception capabilities, deep semantic parsing capabilities, and autonomous behavior planning capabilities. Summary of the Invention

[0004] The purpose of this invention is to address the problem that existing service robot technologies cannot meet the needs of natural language interaction in complex scenarios, nor can they achieve accurate understanding of task intent and autonomous decision-making. This invention provides a method and system for robot voice interaction and autonomous behavior decision-making based on multimodal perception.

[0005] A method for robot voice interaction and autonomous behavior decision-making based on multimodal perception, comprising:

[0006] S1. Record the speech and preprocess it into an audio file, then convert the audio file into text and calculate the speech confidence of the text.

[0007] S2. Determine whether the voice confidence is less than the first preset confidence threshold. If the result is yes, proceed to step S1. If the result is no, proceed to step S3.

[0008] S3. The large language model performs structured semantic parsing of the text: Based on each action instruction in the text and the execution parameters corresponding to that action instruction, it generates structured data and confidence scores in JSON format corresponding to that action instruction; and after weighted averaging of the confidence scores corresponding to all action instructions to obtain the instruction parsing confidence score, it executes step S4.

[0009] S4. Weightedly fuse the instruction parsing confidence score and the speech confidence score to obtain the basic confidence score, and then proceed to step S5.

[0010] S5. Determine whether multimodal verification is needed based on the content described in the text. If the result is yes, proceed to step S6. If the result is no, proceed to step S7.

[0011] S6. Calculate the visual confidence of the target entity described in the text and the map location confidence of the reference location, and then weight and fuse the visual confidence, map location confidence and basic confidence to obtain the final weighted confidence. Then proceed to step S8.

[0012] S7. Use the base confidence level as the final weighted confidence level and proceed to step S8;

[0013] S8. Determine whether the final weighted confidence level is greater than the second preset confidence threshold. If the result is yes, execute the corresponding action according to the structured action data in each JSON format. If the result is no, return to step S1.

[0014] Preferably, in step S1, the method for converting the audio file into text and calculating the speech confidence of the text includes:

[0015] The cloud-based ASR is used to convert audio files into text and output the confidence score of each recognized word in the text. The arithmetic mean of the confidence scores of all words is taken to obtain the speech confidence score of the text.

[0016] Preferably, in step S5, the criterion for determining whether multimodal verification is needed based on the content described in the text is as follows:

[0017] When the text contains descriptions of one or both of the target entity and the target location, multimodal verification is required.

[0018] Preferably, the target entity includes at least one of the target object and the target object attribute; the target location includes at least one of the following: storage area, work area, assembly area, corridor, doorway, corner, behind door, under window sill, and next to machine.

[0019] Preferably, the action instruction includes action keywords and the target entity location;

[0020] The action keywords include forward, backward, left turn, right turn, grab, and release.

[0021] Preferably, the implementation methods for calculating the visual confidence of the target entity described in the text include:

[0022] Obtain the confidence score of the bounding box output by the visual object detection model; obtain the matching degree between the attributes of the visually detected entities and the attributes of the target entities described in the text, as the attribute matching degree; obtain the tracking stability of the visually detected entities in continuous video.

[0023] The visual confidence of the target entity is obtained by weighting the detection box confidence, attribute matching degree and tracking stability.

[0024] Preferably, the visual object detection model is used to output the confidence score of the detection box based on the entities detected visually;

[0025] Furthermore, the detection box confidence score represents the probability that a target exists within the detection box.

[0026] Preferably, the tracking stability is measured based on the continuity and / or trajectory smoothness of visually detected entities being successfully tracked in consecutive video frames.

[0027] Preferably, the implementation methods for calculating the map location reliability of the target location in the environment described in the text include:

[0028] Based on the robot's built-in SLAM algorithm, a 3D map of the scene in which the robot is located is constructed;

[0029] Based on the external LiDAR, scan the 3D scene map of the robot's location;

[0030] The feature point matching method is used to perform consistency feature matching between the coordinates of the target location described in the text in the 3D map and its coordinates in the 3D scene map, and the matching similarity is obtained. This matching similarity is used as the map location reliability of the target location.

[0031] A robot voice interaction and autonomous behavior decision-making system based on multimodal perception includes a storage device, a processor, and a computer program stored in the storage device and executable on the processor. The processor executes the computer program to implement the robot voice interaction and autonomous behavior decision-making method based on multimodal perception.

[0032] The beneficial effects of this invention are:

[0033] This invention proposes a multimodal perception-based robot voice interaction and autonomous behavior decision-making method, enabling robots to achieve end-to-end autonomous control from natural language to structured action sequences, and then to navigation and task execution. The core innovation of this invention lies in proposing a natural language parsing mechanism based on weighted confidence fusion. This mechanism intelligently decomposes complex sentences, multi-turn instructions, and parameterized instructions by evaluating the confidence levels and weighting and fusing multimodal information such as speech recognition, semantic parsing, visual perception, and localization perception. This allows for accurate understanding of task intent and autonomous decision-making, thereby precisely driving robot actions.

[0034] This invention supports multi-level confidence assessment and multi-modal verification.

[0035] The specific advantages are as follows:

[0036] 1. Multimodal decision-making mechanism based on weighted confidence fusion: By dynamically weighting and fusing multiple sources of confidence such as ASR, rule matching, LLM parsing, visual detection and map localization, the system can achieve intelligent understanding and reliable execution of complex instructions, significantly improving the robustness of the system in noisy and semantically ambiguous scenarios.

[0037] 2. Large Model Parsing: By performing structured parsing on the Large Language Model (LLM), uncontrollable text output is avoided, and structured data in executable JSON format is generated.

[0038] 3. Condition-triggered multimodal verification mechanism: Intelligently determines whether multimodal verification is needed based on text content, avoiding unnecessary computational overhead. Cross-verification of visual and map information is only performed in key scenarios involving visual entities and location navigation, improving system efficiency.

[0039] 4. Confidence-based hierarchical decision-making framework: Establish a complete confidence assessment chain from speech recognition to multimodal fusion to achieve hierarchical decision-making and quality control, ensuring that high-confidence commands are executed quickly and low-confidence commands trigger a confirmation process, thus balancing efficiency and security. Attached Figure Description

[0040] Figure 1 This is a flowchart of the robot voice interaction and autonomous behavior decision-making method based on multimodal perception described in this invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0043] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0044] Specific Implementation Method 1: Combination Figure 1 This embodiment describes a robot voice interaction and autonomous behavior decision-making method based on multimodal perception, which includes:

[0045] S1. Record the speech and preprocess it into an audio file, then convert the audio file into text and calculate the speech confidence of the text.

[0046] S2. Determine whether the voice confidence is less than the first preset confidence threshold. If the result is yes, proceed to step S1. If the result is no, proceed to step S3.

[0047] S3. The large language model performs structured semantic parsing of the text: Based on each action instruction in the text and the execution parameters corresponding to that action instruction, it generates structured data and confidence scores in JSON format for that action instruction.

[0048] Then, after weighted averaging the confidence scores of all action commands to obtain the command parsing confidence score, step S4 is executed.

[0049] Action instructions include action keywords and target entity locations;

[0050] The action keywords include forward, backward, left turn, right turn, grab, and release;

[0051] S4. Weightedly fuse the instruction parsing confidence score and the speech confidence score to obtain the basic confidence score, and then proceed to step S5.

[0052] S5. Determine whether multimodal verification is needed based on the content described in the text. If the result is yes, proceed to step S6. If the result is no, proceed to step S7.

[0053] S6. Calculate the visual confidence score of the target entity described in the text and the map location confidence score of the target location, and then weight and fuse the visual confidence score, map location confidence score and basic confidence score to obtain the final weighted confidence score. Then proceed to step S8.

[0054] S7. Use the base confidence level as the final weighted confidence level and proceed to step S8;

[0055] S8. Determine whether the final weighted confidence level is greater than the second preset confidence threshold. If the result is yes, the lower-level machine executes the corresponding action according to the structured action data in each JSON format. If the result is no, return to step S1.

[0056] Principle Analysis: The method of this invention obtains the instruction text and speech confidence through speech recognition. After initial screening by threshold, the instruction is parsed into a structured JSON action sequence (such as forward, grab, etc.) using a large language model and the parsing confidence is calculated. The two are then combined to form the basic confidence.

[0057] Then, based on the text content, it is intelligently determined whether multimodal verification needs to be introduced: if so, the visual confidence of the target entity (combining the confidence of the detection box, the attribute matching degree and the tracking stability) and the map location confidence of the target location are calculated simultaneously, and then weighted and fused with the basic confidence to generate the final weighted confidence; if not, the basic confidence is directly used as the final result.

[0058] Ultimately, through a confidence threshold decision, the robot's lower-level machine executes the corresponding structured action only when the final weighted confidence level meets the standard; otherwise, it requests input again, thereby achieving reliable and adaptive human-computer interaction and autonomous decision-making.

[0059] This invention constructs a multi-level confidence assessment system that includes speech recognition, semantic parsing, visual perception, and localization, and sets multiple judgments and threshold filtering to effectively filter out low-quality and highly uncertain input commands. This significantly reduces the risk of false triggering and false execution caused by environmental noise, speech ambiguity, or sensor errors, and improves the accuracy of the robot's understanding of the user's true intentions and the reliability of the overall interaction.

[0060] Achieving intelligent multimodal information fusion and collaborative verification: Based on the instruction content (S5), it can intelligently determine whether visual and map positioning information is needed for auxiliary verification (S6), and when necessary, it integrates multi-source heterogeneous perception information (visual confidence, map positioning confidence) for comprehensive decision-making. This on-demand triggering fusion mechanism not only ensures the decision-making quality of complex instruction execution but also avoids unnecessary computational overhead, improving the system's intelligence level and execution efficiency.

[0061] It provides structured, executable, and precise control commands: Utilizing a large language model (S3), natural language commands are parsed into structured JSON format data, transforming ambiguous user speech into parameterized action sequences that the robot's lower-level machine can precisely execute. This not only achieves a direct and accurate mapping from high-level semantics to low-level control but also facilitates the execution, monitoring, and debugging of actions through structured data.

[0062] Constructing a closed-loop, adaptive interactive decision-making process: The method of this invention forms a complete closed loop (S1-S8) of "perception-analysis-verification-decision-execution-feedback". It can adaptively select to execute actions or re-request input based on the confidence assessment results, enhancing the robot's autonomous decision-making ability and task completion success rate in dynamic and unstructured environments, and providing a safe and reliable technical foundation for human-robot collaboration.

[0063] Furthermore, in step S1, the method for converting the audio file into text and calculating the speech confidence of the text includes:

[0064] The cloud-based ASR is used to convert audio files into text and output the confidence score of each recognized word in the text. The arithmetic mean of the confidence scores of all words is taken to obtain the speech confidence score of the text.

[0065] Furthermore, in step S5, the criterion for determining whether multimodal verification is needed based on the content described in the text is as follows:

[0066] When the text contains descriptions of one or both of the target entity and the target location, multimodal verification is required.

[0067] The target entity includes at least one of the target object and the target object's attributes;

[0068] The target location includes at least one of the following: storage area, work area, assembly area, corridor, doorway, corner, behind door, under window sill, and next to machine.

[0069] Furthermore, methods for calculating the visual confidence of the target entity described in the text include:

[0070] Obtain the detection box confidence score output by the visual object detection model; the visual object detection model is used to output the detection box confidence score based on the visually detected entities; and the detection box confidence score represents the probability that an object exists within the detection box; obtain the matching degree between the attributes of the visually detected entities and the attributes of the target entities described in the text, as the attribute matching degree; obtain the tracking stability of the visually detected entities in continuous video; wherein, the tracking stability is measured based on the continuity of successful tracking of the visually detected entities in continuous video frames and / or the trajectory smoothness;

[0071] The visual confidence of the target entity is obtained by weighting the detection box confidence, attribute matching degree and tracking stability.

[0072] This preferred embodiment constructs a multi-dimensional, dynamic visual perception evaluation framework by fusing the target detection model's output bounding box confidence, attribute semantic matching degree, and temporal tracking stability. Its core effect is a significant improvement in the reliability and intelligence level of the robot's recognition and confirmation of specific target entities in the physical world. It no longer relies solely on the detection results of a single frame image, but performs progressive verification from three logical levels: "whether it may exist," "whether it matches the description," and "whether it continuously and stably exists." This design effectively addresses common false detections in complex scenes (such as background interference), semantic ambiguity (such as distinguishing similar objects), and dynamic uncertainties (such as occlusion or motion blur), ensuring that the visual system outputs not just an isolated number, but a comprehensive confidence judgment validated by multi-evidence fusion and temporal context. This provides a solid and interpretable perceptual foundation for subsequent cross-modal weighted fusion with speech and location information to achieve accurate and robust autonomous decision-making.

[0073] Furthermore, methods for calculating the map location reliability of the target location in the environment described in the text include:

[0074] Based on the robot's built-in SLAM algorithm, a 3D map of the scene in which the robot is located is constructed;

[0075] Based on the external LiDAR, scan the 3D scene map of the robot's location;

[0076] The feature point matching method is used to perform consistency feature matching between the coordinates of the target location described in the text in the 3D map and its coordinates in the 3D scene map, and the matching similarity is obtained. This matching similarity is used as the map location reliability of the target location.

[0077] In this preferred embodiment, feature matching between the robot's built-in map and the real-time scene from the external LiDAR device enables the assessment of the reliability and accuracy of target location information in text commands. Its core effect lies in constructing a dynamic and verifiable environmental positioning reliability mechanism, effectively solving the static and outdated problems that may arise from solely relying on prior maps.

[0078] By quantifying and comparing the location in the semantic instructions with the three-dimensional features of the physical world, this method can objectively assess the consistency between "the robot's current perceived location" and "the actual location indicated by the instruction," thereby significantly enhancing the robot's decision-making safety and execution accuracy when executing location-related instructions (such as "go to point A"), and providing key spatial confidence basis for multimodal fusion decision-making.

[0079] Specific Implementation Method Two: The robot voice interaction and autonomous behavior decision-making system based on multimodal perception described in this implementation method includes a storage device, a processor, and a computer program stored in the storage device and executable on the processor. The processor executes the computer program to implement the robot voice interaction and autonomous behavior decision-making method based on multimodal perception as described in Specific Implementation Method One.

[0080] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A method for robot voice interaction and autonomous behavior decision-making based on multimodal perception, characterized in that, The method includes: S1. Record the speech and preprocess it into an audio file, then convert the audio file into text and calculate the speech confidence of the text. S2. Determine whether the voice confidence is less than the first preset confidence threshold. If the result is yes, proceed to step S1. If the result is no, proceed to step S3. S3. The large language model performs structured semantic parsing of the text: Based on each action instruction in the text and the execution parameters corresponding to that action instruction, it generates structured data and confidence scores in JSON format corresponding to that action instruction; and after weighted averaging of the confidence scores corresponding to all action instructions to obtain the instruction parsing confidence score, it executes step S4. S4. Weightedly fuse the instruction parsing confidence score and the speech confidence score to obtain the basic confidence score, and then proceed to step S5. S5. Determine whether multimodal verification is needed based on the content described in the text. If the result is yes, proceed to step S6. If the result is no, proceed to step S7. S6. Calculate the visual confidence of the target entity described in the text and the map location confidence of the reference location, and then weight and fuse the visual confidence, map location confidence and basic confidence to obtain the final weighted confidence. Then proceed to step S8. S7. Use the base confidence level as the final weighted confidence level and proceed to step S8; S8. Determine whether the final weighted confidence level is greater than the second preset confidence threshold. If the result is yes, execute the corresponding action according to the structured action data in each JSON format. If the result is no, return to step S1.

2. The robot voice interaction and autonomous behavior decision-making method based on multimodal perception according to claim 1, characterized in that, In step S1, the method for converting the audio file into text and calculating the speech confidence of the text includes: The cloud-based ASR is used to convert audio files into text and output the confidence score of each recognized word in the text. The arithmetic mean of the confidence scores of all words is taken to obtain the speech confidence score of the text.

3. The robot voice interaction and autonomous behavior decision-making method based on multimodal perception according to claim 1, characterized in that, In step S5, the criterion for determining whether multimodal verification is needed based on the content described in the text is as follows: When the text contains descriptions of one or both of the target entity and the target location, multimodal verification is required.

4. The robot voice interaction and autonomous behavior decision-making method based on multimodal perception according to claim 3, characterized in that, The target entity includes at least one of the target object and the target object attribute; the target location includes at least one of the following: storage area, work area, assembly area, corridor, doorway, corner, behind door, under window sill, and next to machine.

5. The robot voice interaction and autonomous behavior decision-making method based on multimodal perception according to claim 1, characterized in that, Action instructions include action keywords and target entity locations; The action keywords include forward, backward, left turn, right turn, grab, and release.

6. The robot voice interaction and autonomous behavior decision-making method based on multimodal perception according to claim 1, characterized in that, Methods for calculating the visual confidence of a target entity described in text include: Obtain the confidence score of the bounding box output by the visual object detection model; obtain the matching degree between the attributes of the visually detected entities and the attributes of the target entities described in the text, as the attribute matching degree; obtain the tracking stability of the visually detected entities in continuous video. The visual confidence of the target entity is obtained by weighting the detection box confidence, attribute matching degree and tracking stability.

7. The robot voice interaction and autonomous behavior decision-making method based on multimodal perception according to claim 6, characterized in that, The visual object detection model is used to output the confidence score of the detection box based on the entities detected by vision; Furthermore, the detection box confidence score represents the probability that a target exists within the detection box.

8. The robot voice interaction and autonomous behavior decision-making method based on multimodal perception according to claim 6, characterized in that, Tracking stability is measured based on the continuity and / or trajectory smoothness of visually detected entities being successfully tracked in consecutive video frames.

9. The robot voice interaction and autonomous behavior decision-making method based on multimodal perception according to claim 1, characterized in that, Methods for calculating the map-based location reliability of a target location in the environment described in the text include: Based on the robot's built-in SLAM algorithm, a 3D map of the scene in which the robot is located is constructed; Based on the external LiDAR, scan the 3D scene map of the robot's location; The feature point matching method is used to perform consistency feature matching between the coordinates of the target location described in the text in the 3D map and its coordinates in the 3D scene map, and the matching similarity is obtained. This matching similarity is used as the map location reliability of the target location.

10. A robot voice interaction and autonomous behavior decision-making system based on multimodal perception, comprising a storage device, a processor, and a computer program stored in the storage device and executable on the processor, characterized in that, The processor executes a computer program to implement the robot voice interaction and autonomous behavior decision-making method based on multimodal perception as described in any one of claims 1 to 9.

Images