An intelligent robot control system and method based on multi-modal interaction

Abstract

The application discloses a kind of intelligent robot control system and method based on multi-modal interaction, including template meta-programming tool registration unit, the parameter type and parameter quantity of function parameter are deduced using compile-time template parameter;Dual-mode voice interaction unit is used to seamlessly switch between cloud large language model and local large language model according to network status, to process the voice instruction of user;Music dance synchronization unit is used to match and generate synchronized dance action sequence from dance library based on audio fingerprint identification and beat analysis of input music;And virtual handle control unit is used to receive control instruction from dual-mode voice interaction unit or music dance synchronization unit, and drive robot to execute corresponding action through unified control interface.The application has type safety advantage, high availability advantage, flexible adaptation advantage, unified interface advantage and scalability advantage.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to an intelligent robot control system and method based on multimodal interaction. Background Technology

[0002] Traditional robot control systems use preset instruction sets or dedicated control interfaces, requiring users to undergo specialized training to operate the robot, resulting in high learning costs, low interaction efficiency, and poor flexibility.

[0003] In recent years, voice-based robot control has gradually emerged, allowing users to communicate directly with robots using natural language. However, existing systems suffer from the following technical problems:

[0004] 1. Tool registration system issues: Tool registration systems typically use runtime reflection or dynamic binding, which incurs runtime overhead and lacks compile-time type safety checks, making them prone to type mismatch errors at runtime.

[0005] 2. Network dependency issue: Large language model interactive systems typically only support online mode. When the network is interrupted, the service cannot be provided, and a seamless transition cannot be achieved.

[0006] 3. Music matching issues: Music and dance matching systems typically bind preset dance moves to specific music clips, which cannot adapt to any music played by the user, resulting in low matching accuracy.

[0007] 4. Control interface issues: There is a lack of unified standards for robot control interfaces, and there are significant differences in control methods between products from different manufacturers, resulting in insufficient real-time monitoring capabilities. Summary of the Invention

[0008] The purpose of this invention is to provide a technical solution for an intelligent robot control system and method based on multimodal interaction, addressing the shortcomings of existing technologies. It achieves compile-time type checking through template metaprogramming technology, completely eliminating runtime type checking overhead and significantly improving execution efficiency. A dual-mode architecture enables network state adaptation, ensuring service continuity without requiring the user to perceive network state changes. A hybrid matching strategy overcomes the limitations of preset music, enabling dance adaptation to any music, with a dynamic adjustment mechanism ensuring that movements follow music changes in real time. Virtual controller technology lowers the control threshold, the DDS protocol ensures product compatibility, and state monitoring enhances controllability and security.

[0009] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0010] A smart robot control system based on multimodal interaction, characterized in that: it includes...

[0011] The template metaprogramming tool registration unit is used to deduce the parameter types and number of parameters of tool functions using compile-time template parameters;

[0012] A dual-mode voice interaction unit is used to seamlessly switch between a cloud-based large language model and a local large language model based on network conditions to process user voice commands.

[0013] The music and dance synchronization unit is used to match and generate synchronized dance movement sequences from the dance library based on audio fingerprint recognition and beat analysis of the input music.

[0014] And a virtual handle control unit, used to receive control commands from the dual-mode voice interaction unit or the music and dance synchronization unit, and drive the robot to perform corresponding actions through a unified control interface.

[0015] This control system achieves compile-time type checking through template metaprogramming technology, completely eliminating runtime type checking overhead and significantly improving execution efficiency; it achieves network state adaptation through a dual-mode architecture, ensuring service continuity without requiring users to be aware of network state changes; it breaks through the limitations of preset music through a hybrid matching strategy, achieving dance adaptation to any music, and a dynamic adjustment mechanism ensures that movements follow music changes in real time; it lowers the control threshold through virtual controller technology, ensures product compatibility through the DDS protocol, and enhances controllability and security through status monitoring functions.

[0016] A method for controlling an intelligent robot based on multimodal interaction, operating within the aforementioned intelligent robot control system based on multimodal interaction, is characterized by comprising the following steps:

[0017] S1, System Initialization

[0018] S1.1 initNetwork(): Initializes network communication and establishes an SSE client connection with the local interactive server. The SSE client is used to receive audio streams from the speech recognition service.

[0019] S1.2 initModels(): Loads the local large language model;

[0020] S1.3 initHardware(): Initializes the robot hardware interface and establishes a DDS communication connection with the Unitree robot;

[0021] S1.4 initMusicDanceMatcher(): Initializes the music and dance matcher and loads the dance motion library;

[0022] S1.5 registerAgentTools(): Registers tool functions to the template metaprogramming tool registration unit, the tool functions including at least a selection action function for performing dance moves and a stop action function for stopping the current dance move;

[0023] S1.6 startWorkerThreads(): Starts the audio processing thread, music and dance thread, or SSE listening thread;

[0024] S1.7 System initialization complete, entering running state;

[0025] S2. During runtime, the collaboration of multiple modules is managed uniformly through the system orchestrator.

[0026] Distribute the audio stream simultaneously to both the speech processing path and the music detection path, including:

[0027] S2.1 For voice interaction scenarios:

[0028] In the speech processing path, S2.1a receives the audio stream through SSE, performs ASR recognition to obtain text, and inputs the text into the large language model for inference. In online mode, it connects to the cloud large language model via TCP, and in offline mode, it uses the local large language model loaded in step S1.

[0029] S2.1b registers the tool unit through the template metaprogramming tool. When a decision is made to execute a dance move, the action selection function is called. The action selection function further calls the application interface of the virtual handle control unit to stop the current dance move and start a new dance move.

[0030] S2.2 For the scenario of dancing to music:

[0031] In the music detection path, S2.2a performs energy detection on the audio stream to determine whether it is music, processes the audio blocks, accumulates buffered audio, identifies the song and analyzes the beat through the song recognition algorithm;

[0032] S2.2b calls the application programming interface of the virtual hand controller to execute dance movements without going through a large language model, thus achieving low-latency response.

[0033] The present invention, by adopting the above-described technical solution, has the following beneficial effects:

[0034] 1. This invention has type safety advantages: by implementing compile-time type checking through template metaprogramming technology, the overhead of runtime type checking is completely eliminated, and the execution efficiency is significantly improved.

[0035] 2. This invention has the advantage of high availability: it achieves network state adaptation through a dual-mode architecture, ensuring service continuity, and users do not need to be aware of changes in network state.

[0036] 3. The present invention has the advantage of flexible adaptation: it breaks through the limitations of preset music through a hybrid matching strategy, realizes dance adaptation to any music, and the dynamic adjustment mechanism ensures that the movements follow the changes of the music in real time.

[0037] 4. This invention has the advantage of a unified interface: it lowers the control threshold through virtual handle technology, ensures product compatibility through the DDS protocol, and enhances controllability and security through status monitoring.

[0038] 5. This invention has the advantage of scalability: the modular coupling design supports the independent evolution and replacement of each functional module, and can quickly adapt to the needs of different application scenarios. Attached Figure Description

[0039] The present invention will be further described below with reference to the accompanying drawings:

[0040] Figure 1 This is a schematic diagram illustrating the connection and interface calls between modules in an intelligent robot control system and method based on multimodal interaction according to the present invention.

[0041] Figure 2 This is a flowchart of the system initialization process in this invention;

[0042] Figure 3 This is a flowchart of the runtime voice interaction scenario in this invention;

[0043] Figure 4 This is a flowchart of the scene where the user listens to music and dances during runtime in this invention. Detailed Implementation

[0044] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0045] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0046] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0047] It should be noted that the English abbreviations, symbols, and terms appearing in the accompanying drawings of this invention are all standard technical parameters or commonly used abbreviations in the field. To facilitate understanding of the technical solution of this invention, the English terms appearing in the accompanying drawings are explained uniformly below:

[0048]

[0049] like Figure 1 As shown, this invention provides an intelligent robot control system based on multimodal interaction, which includes a template metaprogramming tool registration unit, a dual-mode voice interaction unit, a music and dance synchronization unit, a virtual handle control unit, and a system choreographer.

[0050] The system orchestrator uses G1FullPipeline to uniformly manage the collaborative operation of the template metaprogramming tool registration unit, dual-mode voice interaction unit, music and dance synchronization unit, and virtual handheld control unit.

[0051] The core of the template metaprogramming tool registration unit is to utilize C++17's variadic templates and ifconstexpr feature to complete parameter type deduction and binding at compile time, completely eliminating runtime type checking overhead.

[0052] The following is the implementation code for the key data structures and registration interface.

[0053] 1. Tool description structure and consensus processing function type

[0054] ToolSpec struct and ToolHandler type definitions

[0055] / / Unified utility function signature: input parameter list, output execution result string

[0056] using ToolHandler = std::function <std::string(const std::vector <std::string>&args)>;

[0057] / / Tool description structure

[0058] struct ToolSpec {

[0059] std::string name; / / Tool name (for LLM to call)

[0060] std::string description; / / Description of the tool's functionality (for LLM understanding)

[0061] std::string arguments_schema; / / Parameter type description (for LLM to generate parameters)

[0062] ToolHandler handler; / / Execution function bound at compile time

[0063] };

[0064] This code segment defines the core data structure of the tool registration system: ToolHandler abstracts the string parameter list output by the Large Language Model (LLM) into a std::vector. <std::string>Input and string output; ToolSpec encapsulates the tool name, function description, parameter pattern, and compile-time bound processing function into a complete tool description unit. This design stores tool specifications in a hash table (unordered_map) at runtime, allowing the LLM to call them by name. The underlying scheduling is handled by a fixed-signature ToolHandler, avoiding the overhead of dynamic type resolution and virtual function dispatch required by runtime reflection. It provides a unified storage interface after type erasure for the subsequent template metaprogramming registration mechanism, forming the data foundation for the "compile-time binding, runtime zero reflection" architecture of this invention.

[0065] 2. Compile-time type inference and function call unpacker (Invoker)

[0066] Invoker template struct implementation—enabling compile-time parameter type deduction and unpacking invocation

[0067] / / Template unpacker: unpacks vector <string>The parameters are converted one by one according to the compile-time type sequence and the target function is called.

[0068] template<typename Func, typename... Args>

[0069] struct Invoker {

[0070] static std::string apply(Func func, const std::vector <std::string>&args) {

[0071] if ( args . size ( ) != sizeof ... ( Args )) {

[0072] YHANGOS_LOG_ERROR << "Exception log file: " << sizeof...(Args)

[0073] << ", Line: " << args.size();

[0074] return " Error : Argument count mismatch " ;

[0075] }}

[0076] Return apply_impl ( func , args , std :: index_sequence_for<Args...> {});

[0077] }}

[0078] template<std::size_t... I>

[0079] static std : : string apply_impl ( Func func ,

[0080] const std::vector <std::string>&args,

[0081] std::index_sequence<I...>) {

[0082] using ReturnType = std::invoke_result_t<Func, Args...>;

[0083] if constexpr (std::is_void_v <returntype>) {

[0084] / / The function returns void, and returns a fixed string after being called.

[0085] func(from_string <std::decay_t <args>>(args[I])...);

[0086] return "Success (void return)";

[0087] } else {

[0088] / / The function has a return value, which is converted to a string and returned to the LLM after the function is called.

[0089] return to_string(func(from_string <std::decay_t <args>>(args[I])...));

[0090] }

[0091] }

[0092] };

[0093] This code snippet implements the core mechanisms for compile-time parameter type inference and runtime parameter unpacking and invocation. (Invoker)<Func, Args...> Using C++17 variadic templates, the parameter type sequence Args... of the target function is determined at compile time; at runtime, the number of parameters is verified by comparing sizeof...(Args) with the actual parameter count, and then std::index_sequence and if constexpr are used to sequentially populate std::vector<int>...>... <string>The string is converted to its corresponding C++ type and the target function is called, while differentiating between void and non-void return values. This mechanism completes all type checks at compile time, and throws an error during compilation if the parameter types do not match, completely eliminating the overhead of dynamic type checking in traditional runtime reflection schemes. This achieves the technical feature of "deriving the parameter types and number of parameters of the tool function using compile-time template parameters" in this invention.

[0094] 3. Unified registration interface registerTool template function

[0095] ToolRegistry::registerTool template function implementation

[0096] / / Templated registration interface: Supports functions with any parameter type (applicable to 0 to N parameters)

[0097] template<typename Func, typename... Args>

[0098] void registerTool(const std::string &name,

[0099] const std::string &description,

[0100] const std::string &schema,

[0101] Func &&func) {

[0102] / / Create a wrapper lambda, and convert the vector <string>Mapping to the objective function

[0103] ToolHandler wrapper = [func = std::forward <func>(func)]

[0104] (const std::vector <std::string>&args) -> std::string {

[0105] return Invoker<Func, Args...> ::apply(func, args);

[0106] };

[0107] ToolSpec spec{name, description, schema, wrapper};

[0108] registerToolRaw(spec);

[0109] }

[0110] / / Specialization for ordinary function pointers: Automatically extract the return type and parameter type list

[0111] template<typename Ret, typename... Args>

[0112] void registerTool(const std::string &name,

[0113] const std::string &description,

[0114] const std::string &schema,

[0115] Ret (*func)(Args...)) {

[0116] registerTool<Ret (*)(Args...), Args...> (name, description, schema,std::move(func));}

[0117] This code snippet provides a unified registration entry point for arbitrary function signatures (0 to N parameters). During registration, a Lambda wrapper forwards the string parameter vector passed from the LLM to the Invoker.<Func, Args...> The `::apply` function performs type conversion and invocation. For ordinary function pointers, a special overload `Ret (*)(Args...)` is provided, which automatically extracts the return type and parameter type list from the compiler, avoiding the need to manually specify template parameters. After registration, `registerToolRaw` is called to write the tool to the tool table. This two-layer template interface allows developers to register tools with a single line of code, eliminating the need to write repeated registration logic for different numbers of parameters. This significantly improves the efficiency of robot skill expansion and ensures that new tools undergo type safety verification at compile time.

[0118] 4. Tool Registration and Invocation Examples

[0119] Practical application of registering dance control utility functions in a robot system

[0120] / / Define dance motion utility functions

[0121] std::string choice_embodied_action(std::string action_name) {

[0122] / / Specific logic implementation

[0123] }

[0124] std::string stop_embodied_action() {

[0125] / / Specific logic implementation

[0126] }

[0127] / / Register the above utility functions with the Agent

[0128] Agent agent(llm_engine);

[0129] agent.session().setSystemPrompt("You are the G1 robot assistant, and you can control the robot to perform dance moves.");

[0130] / / Register single-parameter tool: The compiler automatically infers Args = {std::string}

[0131] agent.registerTool("choice_embodied_action",

[0132] "Perform the specified dance moves",

[0133] "action_name (string): Dance move name",

[0134] choice_embodied_action);

[0135] / / Register parameterless tools: compiler inference of Args = {}

[0136] agent.registerTool("stop_embodied_action",

[0137] "Stop the currently performed dance move",

[0138] "No parameters"

[0139] stop_embodied_action);

[0140] This code snippet demonstrates a typical application of template metaprogramming tool registration in a robot dance control scenario: registering embodied action selection and stop control functions as tools that can be called by the agent. When registering "choice_embodied_action", the compiler automatically infers Args = {std::string}, and when registering "stop_embodied_action", it infers Args = {} (no parameters). The LLM can then select and call the appropriate tool based on the user's natural language intent during dialogue inference. This example implements a complete closed loop of "natural language understanding → tool selection and parameter generation → robot action execution," demonstrating the technical solution of this invention that deeply integrates template metaprogramming tool registration with multimodal robot control, enabling even non-professional users to drive the robot to perform specified dance movements via voice commands.

[0141] Compared to traditional runtime reflection registration schemes, the template metaprogramming-based scheme described above completes all type checking and function binding at compile time, and the runtime call path contains no virtual functions or type conversion overhead. Table 1 compares the key indices of the two schemes.

[0142] Table 1

[0143] The template metaprogramming tool registration unit includes: a parameter type deduction module, a type conversion module, a function signature storage module, and a parameter binding module.

[0144] The parameter type deduction module utilizes variable parameter template technology to provide a unified registration interface for utility functions with different numbers and types of parameters. This eliminates the need to write separate registration code for functions with different numbers of parameters.

[0145] The type conversion module is used to automatically handle conversion relationships between different types at compile time. It provides implicit type conversion capabilities for basic types and custom conversion logic for complex types.

[0146] The function signature storage module is used to store the names, return types, and parameter type sequences of registered utility functions using type list technology.

[0147] The parameter binding module is used to bind the passed parameter values ​​to the corresponding formal parameters in the order of the function signature, and to generate a compilation error when the types are incompatible.

[0148] The dual-mode voice interaction unit is used to seamlessly switch between a cloud-based large language model and a local large language model based on network conditions to process user voice commands.

[0149] The dual-mode voice interaction unit includes: a network status monitoring module, a dual-mode switching control module, a context management module, and a streaming response processing module.

[0150] The core of the dual-mode voice interaction unit is the seamless switching between online and offline LLM inference modes, as well as the streaming ASR speech recognition pipeline. In online mode, a persistent TCP connection is used to connect to a remote server, while in offline mode, the L1mEngine is used locally.

[0151] The key implementation code is given below:

[0152] 1. ASR Engine Initialization Configuration

[0153] ASREngine Initialization Configuration - Transducer Streaming Recognition Mode

[0154] / / ASR engine initialization (non-streaming mode, based on ONNX Transducer architecture)

[0155] bool initASR() {

[0156] ASRConfig cfg;

[0157] cfg.use_streaming = false; / / Offline recognition: Batch recognition after complete audio segment recording

[0158] cfg.model_dir = resolveModelPath(params_.asr_model_dir);

[0159] / / Transducer Decoder:Encoder / Decoder / Joiner

[0160] cfg.transducer_encoder = cfg.model_dir + " / encoder.onnx";

[0161] cfg.transducer_decoder = cfg.model_dir + " / decoder.onnx";

[0162] cfg.transducer_joiner = cfg.model_dir + " / joiner.onnx";

[0163] cfg.tokens_file = cfg.model_dir + " / tokens.txt";

[0164] cfg.num_threads = params_.asr_num_threads; / / spell2 setting

[0165] asr_ = std::make_unique <asrengine>();

[0166] return asr_->Initialize(cfg) == StatusCode::SUCCESS;

[0167] }

[0168] / / Submit ASR recognition after the speech segment is completed

[0169] void runASRAndSend(const std::vector <float>&audio) {

[0170] AudioData wav;

[0171] wav.samples = audio;

[0172] wav.sample_rate = 16000;

[0173] wav.num_channels = 1;

[0174] ASRResult res;

[0175] if (asr_->Recognize(wav, res) != StatusCode::SUCCESS) {

[0176] YHANGOS_LOG_WARNING << "ASR recognition failed";

[0177] return;

[0178] }

[0179] / / Remove leading and trailing whitespace

[0180] std::string text = res.text;

[0181] size_t first = text.find_first_not_of(" \t\r\n");

[0182] if (first == std::string::npos) return;

[0183] text = text.substr(first, text.find_last_not_of(" \t\r\n") - first +1);

[0184] YHANGOS_LOG_INFO << "Recognition result: " << text << " (RTF=" << res.rtf <<")";

[0185] }

[0186] This code snippet implements an offline speech recognition pipeline based on the ONNX Transducer architecture (Encoder / Decoder / Joiner suite). `initASR` handles model path parsing, Transducer component loading, and inference thread configuration; `runASRAndSend` encapsulates the acquired 16kHz mono floating-point audio into an `AudioData` structure, calls `ASREngine::Recognize` to perform batch recognition, and prunes the recognized text before outputting it. This non-streaming recognition strategy is suitable for high-precision transcription scenarios after complete speech segment recording, providing a unified text input for subsequent online / offline LLM inference, and forming a key front-end module for "speech → text" in a dual-mode speech interaction unit.

[0187] 2. VAD+RMS auxiliary detection – to prevent accidental segmentation of long sentences.

[0188] RMS energy calculation and auxiliary VAD logic

[0189] / / RMS energy calculation function (high real-time requirements, implemented using pure arithmetic)

[0190] static float calculateRMS(const std::vector <float>&samples) {

[0191] if (samples.empty()) return 0.0f;

[0192] float sum_sq = 0.0f;

[0193] for (float s : samples) sum_sq += s * s;

[0194] return std::sqrt(sum_sq / samples.size());

[0195] }

[0196] / / VAD + RMS dual detection logic in the main audio processing loop

[0197] void audioProcessingLoop() {

[0198] std::vector <float>chunk

[0199] while (running_) {

[0200] if (!audio_queue_->pop(chunk)) continue;

[0201] if (playing_prompt_.load()) continue; / / Ignore the audio during the play-prompt.

[0202] float rms = calculateRMS(chunk);

[0203] bool is_energy_high = (rms > params_.audio_energy_threshold); / / Default threshold 0.005

[0204] / / VAD processing

[0205] vad_->AcceptAudioChunk(chunk.data(), chunk.size(), params_.sample_rate);

[0206] std::vector <speechsegment>segs;

[0207] bool has_voice_detected = (vad_->GetDetectedSegments(segs) > 0);

[0208] / / Critical Fix: When VAD has no voice but high energy, it is assumed to still be speaking, and the mute timer is reset.

[0209] / / Resolves the issue of long sentences being misinterpreted as silent and thus cut off when VAD does not output a segment.

[0210] if (seg_state_.in_segment && !has_voice_detected && is_energy_high) {

[0211] seg_state_.last_voice_tp = std::chrono::steady_clock::now();

[0212] }

[0213] / / ... Subsequent mute timeout judgment logic

[0214] }

[0215] }

[0216] This code snippet implements a collaborative mechanism between VAD (Voice Activity Detection) and RMS energy detection to address the issue of accidental segmentation of long speech sentences. `calculateRMS` uses pure arithmetic to perform real-time root mean square energy calculation, meeting the low latency requirements of the audio processing thread. The `audioProcessingLoop` executes VAD detection and RMS threshold judgment (default threshold 0.005) in parallel on each audio block. When it detects that "VAD has not output a speech segment but the RMS energy is still higher than the threshold," it resets the silence timer to prevent brief silences in the middle of long sentences from being mistakenly interpreted as the end of the sentence. This dual detection strategy significantly improves the complete capture rate of continuous speech commands. Simultaneously, this loop also diverts high-energy non-speech segments to the music detection queue, achieving parallel distribution of the two processing paths: speech interaction and music detection. This embodies the technical feature of "dual-path audio stream distribution" in the system's orchestrator.

[0217] 3. Online / offline dual-mode switching and LLM streaming inference

[0218] L1mEngine local inference interface

[0219] / / Streaming output callback function type: Aborts generation when it returns false.

[0220] using StreamCallback = std::function<bool(const std::string &piece)> ;

[0221] class LlmEngine {

[0222] public:

[0223] / / Load local quantization models (supports Qwen / ChatGLM, etc.)

[0224] bool loadModel(const std::string &model_path,

[0225] int32_t n_ctx = 65535, / / Context window length

[0226] int32_t n_threads = 4); / / Number of inference threads

[0227] / / Synchronous inference: Returns the complete generated text

[0228] std::string generate(const std::string &prompt,

[0229] int max_new_tokens = 11024,

[0230] float temperature = 0.8f,

[0231] float top_p = 0.9f,

[0232] int top_k = 40);

[0233] / / Streaming inference: token-based callback with real-time interruption support

[0234] void streamGenerate(const std::string &prompt,

[0235] StreamCallback callback,

[0236] int max_new_tokens = 11024,

[0237] float temperature = 0.8f,

[0238] float top_p = 0.9f,

[0239] int top_k = 40);

[0240] };

[0241] / / Example of online mode call (connecting to a remote LLM service via TCP port 9011)

[0242] json task;

[0243] task["query"] = recognized_text; / / ASR recognition result

[0244] task["task_type"] = "task";

[0245] task["agent"] = "Agent";

[0246] task["session_id"] = "g1_session";

[0247] client_->sendTaskResult(task); / / Send to the remote end, waiting for SSE streaming response.

[0248] / / Example of offline mode call (local LlmEngine streaming inference)

[0249] agent.streamChat(recognized_text, [](const std::string &piece) {

[0250] / / Send the generated segments to TTS in real time

[0251] tts_engine.Synthesize(piece, tts_result);

[0252] return true; / / Continue generating

[0253] }, 256, 0.7f);

[0254] This code segment defines the core interface of the local large language model inference engine and provides calling paradigms for both online and offline working modes. In offline mode, the local quantization model based on llama.cpp (supporting Qwen / ChatGLM, etc.) is loaded via LlmEngine::loadModel, and token-level streaming generation is achieved using streamGenerate / streamChat. The generated fragments are pushed to TTS synthesis in real time, and generation can be interrupted by returning false via a callback. In online mode, ASR-recognized text is uploaded to the cloud LLM service via a TCP task channel (port 9011) using YhangOSClient::sendTaskResult, and the cloud results are returned streaming via SSE. The system automatically switches between the two modes based on network connection status. In the event of a network interruption, it seamlessly degrades to the local Agent + local TTS, without the user being aware of the switching process. This achieves the technical effect of "dual-mode architecture to achieve network state adaptation and ensure service continuity" as described in this invention.

[0255] Table 2 presents a comparison of key performance indicators for the dual-mode system under different network conditions. All data are from actual measurements taken on the Unitree Robotics G1 robot.

[0256] Table 2

[0257] The network status monitoring module is used to continuously detect network connectivity and communication quality by periodically sending heartbeat packets and measuring response latency.

[0258] The dual-mode switching control module is used to seamlessly switch between the cloud-based large language model and the local large language model, so as to achieve the switching between online mode and offline mode working states.

[0259] The context management module is used to maintain dialogue context information and ensure the continuity of dialogue history during mode switching.

[0260] The streaming response processing module is used to support the SSE protocol to receive streaming output of large language models and realize the real-time presentation of response content.

[0261] The music and dance synchronization unit is used to match and generate synchronized dance movement sequences from a dance library based on audio fingerprint recognition and beat analysis of the input music.

[0262] The music and dance synchronization unit adopts a hybrid approach of "cloud recognition + beat analysis" and achieves dance adaptation to any music through a three-level priority matching strategy.

[0263] The following is the core implementation code:

[0264] 1. Three-level priority dance matching algorithm

[0265] C++ Implementation of a Three-Level Priority Matching Algorithm

[0266] / / Level 3 priority dance matching function

[0267] / / target_bpm: The analyzed music BPM

[0268] / / song_name: The name of the identified song (can be empty)

[0269] DanceInfo findBestDance(float target_bpm, const std::string &song_name) {

[0270] / / ---- First priority: Fuzzy matching of song names ----

[0271] if (!song_name.empty()) {

[0272] / / Specific logic implementation

[0273] YHANGOS_LOG_INFO << "First priority hit: ";

[0274] return dance;

[0275] }

[0276] / / ---- Second Priority: BPM Range Filtering + L1 Distance Scoring ----

[0277] float bpm_range = xxx.0f; / / Search range

[0278] std::vector <danceinfo>candidates

[0279] for (const auto &dance : dance_library_) {

[0280] if (std::abs(dance.bpm - target_bpm) <= bpm_range)

[0281] candidates.push_back(dance);

[0282] }

[0283] if (!candidates.empty()) {

[0284] / / score = |candidate_bpm - target_bpm| (L1 distance / Manhattan distance)

[0285] / / Specific logic implementation

[0286] YHANGOS_LOG_INFO << "Second priority hit: ";

[0287] return *best;

[0288] }

[0289] / / ---- Third Priority: Nearest Neighbor Coverage for the Entire Database ----

[0290] / / Specific logic implementation

[0291] YHANGOS_LOG_INFO << "Third priority fallback:" ;

[0292] return *best;

[0293] }

[0294] This code snippet implements a mixed matching strategy for arbitrary music, employing a three-level descending priority to ensure matching success rate and accuracy:

[0295] 1. First priority (fuzzy matching of song name): Match the song name identified by audio fingerprint with the dance_music field of the dance library. If a match is found, the corresponding dance is selected directly, skipping the BPM calculation, to achieve zero-delay accurate matching of known songs.

[0296] 2. Second priority (BPM range filtering + L1 distance score): When the first priority is not matched, filter the candidate dances whose BPM falls within the range of [target_bpm±20], and select the minimum value with score=|candidate_bpm-target_bpm|;

[0297] 3. Third priority (nearest neighbor fallback): When there are no candidates for the second priority, the dance with the smallest absolute difference in BPM is selected by traversing the entire dance library.

[0298] This three-level strategy breaks through the limitations of the traditional "preset music - fixed dance" binding, enabling the system to adapt the dance to any music played by the user. In the engineering implementation (MusicDanceMatcher::RecognizeAndMatch), a same song detection mechanism (song name inclusion relationship, BPM identifier matching, and priority for the same dance name) is also integrated to avoid the repeated recognition of the same song, which leads to frequent dance switching. This reflects the technical feature of the invention, "dynamic adjustment mechanism to ensure that the movement follows the music changes in real time".

[0299] Table 3 presents the hit rate statistics of the three-level priority matching strategy in the actual dance database.

[0300] Table 3

[0301] The music and dance synchronization unit includes: an audio fingerprint extraction module, a beat detection module, a dance movement matching module, and a dynamic adjustment module.

[0302] The audio fingerprint extraction module is used to process the input audio and generate an audio fingerprint to identify the song. The processing of the input audio includes frame segmentation (frame length 2048 sampling points, frame shift 1024 sampling points), Fourier transform, peak detection and hash encoding.

[0303] This invention uses a cloud-based recognition algorithm for audio fingerprint matching. The specific implementation process is as follows:

[0304] 1. Audio Acquisition: The C++ layer obtains real-time audio streams from the local interactive server through the SSE (Server-Sent Events) client. The sampling rate is 16kHz and the format is float32 PCM.

[0305] 2. Audio Buffering: After accumulating 5 seconds of audio data, a recognition process is triggered. To improve response speed, a sliding window mechanism is used, sliding once every 1 second, with each analysis window lasting 5 seconds.

[0306] 3. Temporary file generation: Write float32 PCM data to a temporary WAV file (sampling rate 16kHz, single channel, 32-bit floating point).

[0307] 4. Python Bridged Call: The C++ layer calls the Python script via subprocess: python3 audio_recognizer.py –a<wav_file> -m recognize.

[0308] 5. Cloud-based recognition: The Python script uses the recognize(audio_file) interface to upload audio data to the cloud server, extract audio fingerprints, perform matching, and return the song name and artist information.

[0309] 6. Results Analysis: The C++ layer parses the JSON format results returned by the Python script and extracts the song name and artist name.

[0310] The beat detection module is used to detect the BPM value of music by performing audio filtering preprocessing (frequency range of 200-1000Hz), energy calculation and autocorrelation analysis.

[0311] The beat time (BPM) analysis technique of this invention uses the Python librosa library for beat time analysis, and the specific process is as follows:

[0312] 1. Audio feature extraction: Call librosa.onset.onset_strength(y,sr=sr,S=None,hop_length=512), the internal process is: short time Fourier transform (STFT, window length 2048) → Mel filter bank (128 filters) → logarithmic compression → forward difference → summation along the time axis to obtain the onset strength envelope.

[0313] 2. BPM tracking: Call librosa.best.best_track(y=y,sr=sr,onset_envelope=onset_env,hop_length=512), internally use the autocorrelation algorithm to estimate the initial BPM value, and then use dynamic programming / Viterbi algorithm to track the beat.

[0314] 3. Default parameters: sampling rate sr=22050Hz, STFT window length=2048, hop_length=512, number of Mel filters=128, BPM search range 60~200.

[0315] 4. Python bridged call: The C++ layer calls via subprocess: python3 audio_recognizer.py –a<wav_file> -m bpm.

[0316] The dance movement matching module uses a hybrid scoring mechanism to select matching dance movement sequences from the dance library, taking into account factors such as similarity to song style, beat matching, and smoothness of movement transitions.

[0317] The dance movement matching module adopts a three-level priority dance matching strategy, including: first priority, second priority, and third priority.

[0318] The first priority is used to perform fuzzy matching of song names, checking whether the dance_music field of the candidate dance library contains the identified song name (case-insensitive, inclusion is considered a match). If a dance exists in the dance library that matches the song name, the match is successful, and the dance sequence is directly selected without proceeding to the subsequent BPM matching process.

[0319] The second priority (BPM range filtering + L1 mid-distance scoring) is used to filter dance moves with BPM values ​​within a preset range from the dance library when the first priority match fails. It calculates the absolute difference of each candidate dance and selects the dance move sequence with the smallest absolute difference with the BPM value as the matching result.

[0320] The BPM value is within the preset range of [BPM-20, BPM+20].

[0321] Calculate the score for each candidate dance: score = |candidate_bpm - target_bpm|, which is the L1 distance (absolute difference) between candidate_bpm and target_bpm.

[0322] The third priority (global nearest neighbor fallback) is used when there are no candidate dances in the second priority selection. It then traverses the entire dance library and selects the dance sequence with the smallest absolute difference from the BPM.

[0323] The dance movement matching module also includes a song-matching detection mechanism to avoid frequent dance switching caused by repeated recognition of the same song.

[0324] The same song detection mechanism includes at least one of the following: song name string inclusion detection (case-insensitive), BPM identifier matching detection (comparing BPM after rounding to an integer), and priority matching of dances with the same dance name but not BPM identifier.

[0325] The dynamic adjustment module is used to dynamically adjust the speed and amplitude of the action based on real-time analysis results, ensuring that the action is synchronized with the music beat.

[0326] The virtual handle control unit is used to receive control commands from the dual-mode voice interaction unit or the music and dance synchronization unit, and drive the robot to perform corresponding actions through a unified control interface.

[0327] The Virtual Gamepad Controller communicates with the Unitree Robotics G1 robot's underlying architecture via the DDS protocol, providing a unified motion execution interface. Upper-level calls (whether from Agent tools or the music / dance synchronization unit) are all converted into commands via an HTTP interface, thus decoupling the control source.

[0328] The following is the code for the core HTTP interface call logic and action execution flow:

[0329] 1. API Action Interface Call (C++ Side)

[0330] Call the API interface of VirtualGamepadController via HttpClient

[0331] / / The complete process of performing dance moves

[0332] std::string choice_embodied_action(std::string action_name) {

[0333] try {

[0334] / / Step 1: Call / set_policy to set the target action policy

[0335] HttpClient http_client("http: / / 127.0.0.1:5000", 10);

[0336] json set_policy_body;

[0337] set_policy_body["action"] = action_name;

[0338] HttpResponse set_resp = http_client.post(" / set_policy", set_policy_body);

[0339] if (!set_resp.isSuccess()) {

[0340] return "error: set_policy failed, HTTP " +

[0341] std::to_string(set_resp.status_code);

[0342] }

[0343] / / Step 2: Stop the current action (if any) to avoid action conflicts.

[0344] http_client.post(" / stop", json::object());

[0345] std::this_thread::sleep_for(std::chrono::milliseconds(200));

[0346] / / Step 3: Start the new dance moves

[0347] HttpResponse start_resp = http_client.post(" / start", json::object());

[0348] if (start_resp.isSuccess()) {

[0349] YHANGOS_LOG_INFO << "Action started successfully: " << action_name;

[0350] return "success: started action " + action_name;

[0351] }

[0352] return "error: start failed";

[0353] } catch (const std::exception &e) {

[0354] return std::string("error: ") + e.what();

[0355] }

[0356] }

[0357] / / Stop the current action

[0358] std::string stop_embodied_action() {

[0359] HttpClient http_client("http: / / 127.0.0.1:5000", 5);

[0360] HttpResponse resp = http_client.post(" / stop", json::object());

[0361] return resp.isSuccess() ? "success: action stopped" : "error: stopfailed";

[0362] }

[0363] This code snippet implements a unified HTTP action execution interface for the virtual gamepad controller, converting upper-layer control intentions (whether from Agent tool calls or music / dance synchronization units) into a strategy switching sequence executable at the robot's underlying level. `choice_embodied_action` executes in four steps: "set policy ( / set_policy → stop current action ( / stop) → delay 200ms → start new action ( / start)", ensuring conflict-free action switching. `stop_embodied_action` immediately stops the current dance via the ` / stop` interface. In the engineering implementation, `VirtualGamepadController` also provides interfaces such as `SetPolicy`, `StartMimic`, and `SwitchToLoco` through FastAPI, and can optionally initialize the Unitree DDS channel to publish LowState simulated gamepad data at a 500Hz frequency, achieving standardized interface with the G1 robot's underlying communication protocol. This unified interface design allows the voice interaction path and music detection path to share the same action execution layer, reducing the integration complexity of multi-source control commands and reflecting the technical feature of this invention: "a unified control interface drives the robot to execute corresponding actions."

[0364] 2. Initialization and startup process of G1Pipeline system orchestrator.

[0365] G1Pipeline System Initialization Process – Unified Management of the Startup Order of Each Module

[0366] bool G1Pipeline::initialize() {

[0367] / / S1.1 initNetwork: Establishes a connection to the local interactive server

[0368] client_ = std::make_unique <yhangosclient>(

[0369] params_.server_host, params_.token,

[0370] params_.info_port, / / Information channel

[0371] params_.task_port, / / Task channel (ASR result upload / LLM result distribution)

[0372] params_.http_port, / / HTTP configuration interface

[0373] params_.robot_id );

[0375] client_->setTaskCallback([this](const json &task) {

[0376] this->handleServerTask(task); / / Process the TTS text returned by the LLM streaming service.

[0377] });

[0378] if (!client_->start()) return false;

[0379] client_->startPeriodicInfoUpload(30); / / Upload a heartbeat every 30 seconds

[0380] / / S1.2 / 1.3 Retrieve voice configuration and robot information from remote end

[0381] fetchSpeechConfig(); / / Get KWS wake word, TTS configuration, etc.

[0382] fetchRobotInfo(); / / Get robot information

[0383] fetchMapPoints(); / / Get map navigation points

[0384] / / S1.4 / 1.5 Initialize the voice processing module

[0385] if (!initVAD() || !initASR()) return false;

[0386] if (params_.pipeline_mode == PipelineMode::KWS)

[0387] if (!initKWS()) return false;

[0388] / / S1.3 initHardware: Initialize audio capture (microphone)

[0389] AudioCaptureConfig cap_cfg;

[0390] cap_cfg.device_id = params_.device_id;

[0391] cap_cfg.sample_rate = 16000;

[0392] cap_cfg.chunk_size = 1600; / / 100ms per chunk

[0393] capture_ = std::make_unique <audiocapture>(cap_cfg);

[0394] if (!capture_->initialize()) return false;

[0395] / / S1.1 gRPC TTS channel (connected to audio playback service)

[0396] grpc_client_ = std::make_unique <echoclient>(

[0397] params_.grpc_server_address, / / Default 192.168.xx:50051

[0398] params_.grpc_timeout_seconds / / Default 3600s );

[0400] if (!grpc_client_->Connect()) return false;

[0401] / / S1.6 startWorkerThreads: Starts the audio processing threads

[0402] audio_thread_ = std::thread([this] { this->audioProcessingLoop();});

[0403] return true;

[0404] }

[0405] This code segment implements the phased initialization process of the system orchestrator (G1FullPipeline) of this invention, starting each functional module in a strict dependency order:

[0406] 1. S1.1 Network Initialization: Establish YhangOSClient multi-channel connections (information channel, task channel, HTTP configuration interface), register task callbacks to handle LLM streaming TTS returns, and start 30-second periodic heartbeat reporting; initialize the SSE client in parallel to receive interactive_server audio streams and gRPC TTS playback channels;

[0407] 2. S1.2 / 1.3 Model and Configuration Retrieval: Initialize VAD / ASR / KWS and other voice models, and retrieve voice configuration, robot role information and map navigation points from the remote end;

[0408] 3. S1.3 Hardware Initialization: Configure an audio acquisition device with a 16kHz sampling rate and 100ms chunks (chunk_size=1600);

[0409] 4. Startup of S1.6 worker thread: Start the audioProcessingLoop audio processing thread to enable concurrent collaboration of multiple modules at runtime.

[0410] The orchestrator uniformly manages the initialization timing and runtime collaboration of the four core units (template metaprogramming tool registration, dual-mode voice interaction, music and dance synchronization, and virtual handle control), enabling the system to enter the running state after all modules are ready at a single entry point. This reflects the modular coupling and independently evolving overall architecture design of this invention.

[0411] Table 4 shows the motion slot configuration supported by the virtual controller control unit and its typical dance motion mapping, as well as the measured end-to-end latency data for executing motions via the API interface.

[0412] Table 4

[0413] The status monitoring module uploads the robot's status in real time via the YhangOSClient information channel, including joint angles, battery level, and movement speed, with a sampling period of 30 seconds (heartbeat packets). When an abnormal state is detected, the system will issue an alarm via TTS synthesized voice and interrupt the current task.

[0414] The virtual controller control unit includes: a virtual controller mapping module, a command conversion module, a model slot management module, and a status monitoring module.

[0415] The virtual controller mapping module is used to map the input signals of the game controller to the control commands of the robot. It supports user-defined button functions and has multiple preset control templates.

[0416] The instruction conversion module is used to convert control instructions into commands that conform to the target robot's communication protocol, and supports the DDS protocol.

[0417] The model slot management module is used to switch the configuration and dynamics between different slots via a handle, supporting the configuration and dynamic switching of 8 action slots.

[0418] The status monitoring module is used to collect and report the robot's status information in real time, and issue an alarm when the status is abnormal.

[0419] The robot's status information includes battery level, joint angles, and movement speed.

[0420] This control system achieves compile-time type checking through template metaprogramming technology, completely eliminating runtime type checking overhead and significantly improving execution efficiency; it achieves network state adaptation through a dual-mode architecture, ensuring service continuity without requiring users to be aware of network state changes; it breaks through the limitations of preset music through a hybrid matching strategy, achieving dance adaptation to any music, and a dynamic adjustment mechanism ensures that movements follow music changes in real time; it lowers the control threshold through virtual controller technology, ensures product compatibility through the DDS protocol, and enhances controllability and security through status monitoring functions.

[0421] Table 5 summarizes the overall performance data of the four core units proposed in this invention in the integrated test on the Unitree Technology G1 humanoid robot. The test environment is an onboard computing unit (ARM architecture, 8 cores) running Ubuntu 22.04.

[0422] Table 5

[0423] like Figures 2 to 4 As shown, this invention provides a multimodal interaction-based intelligent robot control method, which operates within the aforementioned multimodal interaction-based intelligent robot control system and includes the following steps:

[0424] S1, System Initialization

[0425] S1.1 initNetwork(): Initializes network communication and establishes an SSE client connection with the local interactive server. The SSE client is used to receive audio streams from the speech recognition service.

[0426] S1.2 initModels(): Loads local large language models (based on models such as Qwen / ChatGLM in llama.cpp);

[0427] S1.3 initHardware(): Initializes the robot hardware interface and establishes a DDS communication connection with the Unitree robot;

[0428] S1.4 initMusicDanceMatcher(): Initializes the music and dance matcher and loads the dance motion library;

[0429] S1.5 registerAgentTools(): Registers tool functions with the Agent tool registration system. The tool functions include at least the choice_embodied_action function for performing dance moves and the stop_embodied_action function for stopping the current dance move.

[0430] S1.6 startWorkerThreads(): Starts the audio processing thread, music and dance thread, or SSE listening thread;

[0431] S1.7 System initialization complete, entering running state;

[0432] S2. During runtime, the collaboration of multiple modules is managed uniformly through the system orchestrator.

[0433] Distribute the audio stream simultaneously to both the speech processing path and the music detection path, including:

[0434] S2.1 For voice interaction scenarios:

[0435] In the speech processing path, S2.1a receives the audio stream through SSE, performs ASR recognition to obtain text, and inputs the text into the Large Language Model (LLM) for inference. In online mode, it connects to the cloud-based Large Language Model (LLM) via TCP9011, while in offline mode, it uses the local Large Language Model loaded in step S1.

[0436] S2.1b invokes tools through the Agent tool registration system (Agent::streamChat). When a decision is made to execute a dance move, the action selection function is invoked. The action selection function further invokes the application programming interface of the virtual controller control unit to stop the current dance move and start a new dance move.

[0437] The dual-mode voice interaction unit is responsible for online / offline mode switching decisions, while the template metaprogramming tool registration unit is responsible for tool registration and tool invocation logic.

[0438] The action execution ultimately calls the virtual controller control unit: the agent calls choice_embodied_action → stop the current action → start the new dance (via FastAPI POST / set_policy, / start_mimic).

[0439] S2.2 For the scenario of dancing to music:

[0440] In the music detection path, S2.2a performs energy detection on the audio stream to determine whether it is music, processes the audio blocks (ProcessAudioChunk), accumulates buffered audio, identifies the song through a song recognition algorithm (Shazam recognition), and analyzes the beat (librosa analysis).

[0441] After the S2.2b music and dance synchronization unit recognizes the song, it directly calls the application programming interface (API) of the virtual gamepad controller to execute the dance movements, without going through the Large Language Model (LLM) to achieve low-latency response.

[0442] dance_matcher directly triggers StopMimic+StartMimic, bypassing the Agent dialogue process.

[0443] The template metaprogramming tool registration unit provides tool invocation capabilities for the dual-mode voice interaction unit and the music and dance synchronization unit. The tools include choice_embodied_action, stop_embodied_action, etc.

[0444] The virtual controller control unit is the final action execution layer, receiving control commands from the template metaprogramming tool registration unit (through LLM inference decision-making) and the music and dance synchronization unit (direct invocation).

[0445] The dual-mode voice interaction unit serves as the entry point for user experience, responsible for natural language understanding and dialogue management, and coordinating tool calls from the template metaprogramming tool registration unit.

[0446] The music and dance synchronization unit is the implementation layer of music entertainment functions. When music is detected, it directly drives the virtual controller control unit to execute dance movements.

[0447] The above are merely specific embodiments of the present invention, but the technical features of the present invention are not limited thereto. Any simple changes, equivalent substitutions, or modifications made based on the present invention to achieve substantially the same technical effect are all covered within the protection scope of the present invention.< / echoclient> < / audiocapture> < / yhangosclient> < / danceinfo> < / speechsegment> < / float> < / float> < / float> < / asrengine> < / std::string> < / func> < / string> < / string> < / args> < / args> < / returntype> < / std::string> < / std::string> < / string> < / std::string> < / std::string>

Claims

1. A smart robot control system based on multimodal interaction, characterized in that: include The template metaprogramming tool registration unit is used to deduce the parameter types and number of parameters of tool functions using compile-time template parameters; A dual-mode voice interaction unit is used to seamlessly switch between a cloud-based large language model and a local large language model based on network conditions to process user voice commands. The music and dance synchronization unit is used to match and generate synchronized dance movement sequences from the dance library based on audio fingerprint recognition and beat analysis of the input music. And a virtual handle control unit, used to receive control commands from the dual-mode voice interaction unit or the music and dance synchronization unit, and drive the robot to perform corresponding actions through a unified control interface.

2. The intelligent robot control system based on multimodal interaction according to claim 1, characterized in that: The template metaprogramming tool registration unit includes: The parameter type deduction module is used to provide a unified registration interface for utility functions with different numbers and types of parameters by utilizing variable parameter template technology; The type conversion module is used to automatically handle conversion relationships between different types at compile time: The function signature storage module is used to store the names, return types, and parameter type sequences of registered utility functions using type list technology; And a parameter binding module, which binds the passed parameter values ​​to the corresponding formal parameters in the order of the function signature, and generates a compilation error when the types are incompatible.

3. The intelligent robot control system based on multimodal interaction according to claim 1, characterized in that: The dual-mode voice interaction unit includes: The network status monitoring module is used to continuously detect network connection status and communication quality by periodically sending heartbeat packets and measuring response latency; The dual-mode switching control module is used to seamlessly switch between the cloud-based large language model and the local large language model, so as to realize the switching between online mode and offline mode working states. The context management module is used to maintain dialogue context information and ensure the continuity of dialogue history during mode switching. It also includes a streaming response processing module, which supports the SSE protocol to receive streaming output of large language models and enable real-time presentation of response content.

4. The intelligent robot control system based on multimodal interaction according to claim 1, characterized in that: The music and dance synchronization unit includes: The audio fingerprint extraction module is used to process the input audio and generate an audio fingerprint to identify the song. The beat detection module is used to detect the BPM value of music by performing filtering preprocessing, energy calculation and autocorrelation analysis on the audio. The dance movement matching module uses a hybrid scoring mechanism to select matching dance movement sequences from the dance library, taking into account factors such as similarity to song style, beat matching, and smoothness of movement transitions. It also includes a dynamic adjustment module, which dynamically adjusts the speed and amplitude of the action based on real-time analysis results to ensure that the action is synchronized with the music beat.

5. The intelligent robot control system based on multimodal interaction according to claim 4, characterized in that: The input audio is processed, including frame segmentation, Fourier transform, peak detection, and hash encoding.

6. The intelligent robot control system based on multimodal interaction according to claim 4, characterized in that: The dance movement matching module employs a three-level priority dance matching strategy, including: The first priority is used to perform fuzzy matching of song names. If a dance that matches the song name exists in the dance library, the match is successful and the dance sequence is selected directly. The second priority is used to filter dance moves with BPM values ​​within a preset range from the dance library when the first priority fails to match. The absolute difference of each candidate dance is calculated, and the dance move sequence with the smallest absolute difference with the BPM value is selected as the matching result. And a third priority, used when there are no candidate dances in the second priority filtering, to traverse the entire dance library and select the dance sequence with the smallest absolute difference from BPM.

7. The intelligent robot control system based on multimodal interaction according to claim 6, characterized in that: The dance movement matching module also includes a song detection mechanism to avoid repeated recognition of the same song, which would lead to frequent dance switching.

8. The intelligent robot control system based on multimodal interaction according to claim 7, characterized in that: The same song detection mechanism includes at least one of the following: song name string inclusion detection, BPM identifier matching detection, and priority matching of dances with the same dance name but not BPM identifier.

9. The intelligent robot control system based on multimodal interaction according to claim 1, characterized in that: The virtual controller control unit includes: The virtual controller mapping module is used to map the input signals of the game controller into control commands for the robot. The instruction conversion module is used to convert the control instructions into commands that conform to the target robot's communication protocol; The model slot management module is used to switch the configuration and dynamics between different slots via a handle; It also includes a status monitoring module, which collects and reports the robot's status information in real time and issues an alarm when the status is abnormal.

10. A multimodal interaction-based intelligent robot control system according to claim 9, characterized in that: The robot's status information includes battery level, joint angles, and movement speed.

11. The intelligent robot control system based on multimodal interaction according to claim 1, characterized in that: It also includes a system orchestrator, which uses G1FullPipeline to uniformly manage the cooperation between the template metaprogramming tool registration unit, the dual-mode voice interaction unit, the music and dance synchronization unit, and the virtual controller control unit during operation.

12. A method for controlling an intelligent robot based on multimodal interaction, operating in an intelligent robot control system based on multimodal interaction as described in any one of claims 1 to 11, characterized in that... The steps include the following: S1, System Initialization S1.1 initNetwork(): Initializes network communication and establishes an SSE client connection with the local interactive server. The SSE client is used to receive audio streams from the speech recognition service. S1.2 initModels(): Loads the local large language model; S1.3 initHardware(): Initializes the robot hardware interface and establishes a DDS communication connection with the Unitree robot; S1.4 initMusicDanceMatcher(): Initializes the music and dance matcher and loads the dance motion library; S1.5 registerAgentTools(): Registers tool functions to the template metaprogramming tool registration unit, the tool functions including at least a selection action function for performing dance moves and a stop action function for stopping the current dance move; S1.6 startWorkerThreads(): Starts the audio processing thread, music and dance thread, or SSE listening thread; S1.7 System initialization complete, entering running state; S2. During runtime, the collaboration of multiple modules is managed uniformly through the system orchestrator. Distribute the audio stream simultaneously to both the speech processing path and the music detection path, including: S2.1 For voice interaction scenarios: S2.1a In the speech processing path, the audio stream is received through SSE, ASR recognition is performed to obtain text, and the text is input into the large language model for inference. In online mode, the large language model in the cloud is connected via TCP, and in offline mode, the local large language model loaded in step S1 is used. S2.1b uses the template metaprogramming tool registration unit to call tools. When a decision is made to execute a dance move, the action selection function is called. The action selection function further calls the application programming interface of the virtual handle control unit to stop the current dance move and start a new dance move. S2.2 For the scenario of dancing to music: S2.2a In the music detection path, energy detection is performed on the audio stream to determine whether it is music, the audio blocks are processed, the buffered audio is accumulated, and the song is identified and the beat is analyzed through the song recognition algorithm. S2.2b calls the application programming interface of the virtual hand controller to execute dance movements without going through a large language model, thus achieving low-latency response.

Images