A Multimodal Human-Computer Voice Interaction Method and System Based on Embodied Intelligence

Abstract

This invention relates to the field of human-computer voice interaction technology, and discloses a multimodal human-computer voice interaction method and system based on embodied intelligence. The system includes a voice interaction module, an audio playback control module, an application adaptation module, and a configuration management module. By designing a lock-free concurrent sublayer with a circular buffer in the audio playback control module, an independent buffer is created for each audio track. Lock-free enqueueing and dequeueing of audio tasks are achieved through independent read / write pointers, completely eliminating lock contention. Furthermore, a lightweight fault-tolerant verification unit can be configured to perform legality checks on the buffer at a set frequency, automatically repairing issues such as pointer out-of-bounds errors and abnormal states. This ensures the continuous and stable operation of the system in a multi-threaded environment from the bottom layer, solving the resource conflicts and concurrency stability problems caused by mutex locks and atomic operations in existing designs.

Description

Technical Field

[0001] This invention relates to the field of human-computer voice interaction technology, and more specifically to a multimodal human-computer voice interaction method and system based on embodied intelligence. Background Technology

[0002] In current human-computer interaction technology, voice interaction, as one of the core interaction methods, has been widely used in many fields such as robots, exhibition hall equipment, and industrial control. It revolves around technical systems such as voice signal acquisition and preprocessing, semantic understanding and synthesis, system deployment architecture, and multimodal linkage, and is adapted to scenarios based on industrial application needs. However, due to limitations in computing power, algorithms, and architecture design, it has significant technical shortcomings in terms of stability, scenario adaptability, and multi-task collaboration.

[0003] Currently, the main processing methods for human-machine voice interaction in embodied intelligent devices (such as service robots, industrial robots, and exhibition hall equipment) include voice signal acquisition, ASR fusion lightweight model recognition, and TTS speech synthesis. Through cloud-based SaaS / local model deployment, complex corpora are expanded and models are iterated in the cloud. Through the fusion of language, vision, touch, and action, after the voice interaction module outputs a command, the vision module (camera) locks onto the speaker, the touch module (sensor) perceives the environment, and the action control module executes mechanical operations. For example, an industrial robot receives operation tasks through voice commands, and the vision module locates parts and the robotic arm completes the assembly. After being woken up by voice, the exhibition hall robot uses the vision module to identify the direction of the visitor's gaze and plays explanatory audio.

[0004] Currently, the integration and collaborative operation of human-computer voice interaction generally adopts linear logic, supporting the serial execution of single or a small number of tasks. This results in high collaborative latency and an inability to handle multiple audio tasks simultaneously (such as a robot simultaneously performing obstacle avoidance broadcasts and facial recognition prompts). Furthermore, the voice module communication uses traditional thread lock mechanisms. In multi-task concurrent scenarios, the use of mutex locks and atomic operations leads to lock contention under multi-threaded conditions, causing audio stuttering, command delays, poor system stability, and susceptibility to data transmission stuttering and command execution delays. In particular, in highly dynamic mobile environments (such as when a robot is moving), there is a risk of voice command delays due to lock contention. Summary of the Invention

[0005] Aiming at the defects and deficiencies of the prior art, the purpose of the present invention is to provide a multimodal human-machine voice interaction system and method based on embodied intelligence. By adopting an end-to-end modular architecture, inter-process communication is achieved between modules through ROS2 topics / services, with low data transmission latency. An unlocked concurrent sub-layer of a circular buffer is integrated in the audio playback control module, creating an independent buffer for each audio track, and implementing lock-free enqueueing and dequeueing of audio tasks through independent read and write pointers. The lock competition and internal scheduling latency are eliminated through the lock-free concurrent mechanism, improving the linkage response between voice interaction and multimodal actions and meeting the real-time requirements of embodied intelligence.

[0006] According to the first aspect of the purpose of the present invention, a multimodal human-machine voice interaction system based on embodied intelligence is proposed, including a voice interaction module, an audio playback control module, an application adaptation module, and a configuration management module; The voice interaction module is used for voice wake-up, speech recognition, semantic understanding, and speech synthesis, and outputs voice interaction results and control instructions to the audio playback control module; The application adaptation module presets adaptation interfaces for multiple application scenarios, identifies valid scenario trigger events based on robot sensor data, and converts them into standardized audio tasks for output to the audio playback control module; The audio playback control module is used to judge the task priority and allocate audio tracks according to the standardized audio tasks submitted by the application adaptation module and the voice interaction module, locate to the circular buffer corresponding to the audio track, and control the single-track controller for playback and management by calling the lock-free enqueue / dequeue logic; The configuration management module analyzes and manages system parameters through a YAML configuration file, including the management of audio parameters, voice interaction parameters, task priority parameters, and lock-free concurrent parameters; Among them, the audio playback control module is configured with a circular buffer lock-free concurrent unit, creating independent circular buffers for multiple audio tracks respectively. Each circular buffer is bound to a single-track controller one by one, so that each track controller can only operate on the read pointer and tasks of the corresponding buffer. The audio playback control module completes the lock-free writing / reading of audio tasks by scheduling the circular buffer lock-free concurrent unit based on the circular movement of the write / read pointer, realizing the lock-free enqueueing and dequeueing of audio task scheduling control.

[0007] As an optional implementation manner, the audio playback control module includes an audio controller, a single-track controller, a standardized encapsulation component, and a circular buffer lock-free concurrent unit; The audio controller operates in singleton mode and is used to receive standardized audio tasks submitted by the application adaptation module and the voice interaction module. It performs task priority judgment, audio track allocation, locates the circular buffer corresponding to the audio track and uses it as the target buffer. It controls the single audio track controller corresponding to the target buffer for playback and management by calling the lock-free enqueue / dequeue logic. The single-track controller is configured with multiple instances according to the robot scenario, and each instance is bound to a corresponding physical track. Each instance corresponds to one track / physical channel. The single-track controller receives scheduling instructions from the audio controller to complete the playback, pause, stop, volume adjustment and playback status management of the single track, and records the playback progress of the audio task to provide support for breakpoint recovery. The standardized encapsulation component is used for the standardized encapsulation of audio tasks, and respectively implements the definition of basic audio task interfaces, the encapsulation of local audio file playback tasks, and the encapsulation of streaming audio playback tasks. The lock-free concurrent unit of the circular buffer is used to independently operate the write pointer of the target buffer according to the circular buffer located by the audio controller, and to cooperate with the single track controller to independently operate the read pointer of the target buffer, so as to realize the lock-free enqueueing, lock-free dequeueing and playback of audio tasks.

[0008] As an optional implementation, the audio controller presets audio task priority levels, for example, setting four priority levels: Level 1 - Emergency, Level 2 - Interactive, Level 3 - Explanation, and Level 4 - Welcome. Simultaneously, based on task type and scene attributes, it assigns four core audio tracks from the service robot (welcome track, explanation track, emergency alert track, and interactive track) to each task, achieving task-track matching. When a high-priority task enters the task queue, it controls the exclusive playback scheduling of the high-priority task and records the playback progress of low-priority tasks. The audio controller triggers an automatic recovery mechanism after a high-priority task is completed, instructing each single-track controller to resume playback of the corresponding low-priority task according to the recorded playback progress, thereby achieving breakpoint recovery control for low-priority tasks.

[0009] As an optional implementation, the ring buffer lock-free concurrent unit includes a buffer management unit, a lock-free enqueue unit, and a lock-free dequeue unit; The buffer management unit is used to create independent circular buffers for multiple audio tracks. Each circular buffer is bound to a single audio track controller, so that each audio track controller can only operate the read pointer and task of the corresponding buffer. The lock-free enqueue unit is used to receive audio tasks assigned by the audio controller. It realizes lock-free writing of audio tasks by independently operating the write pointer of the target buffer. After writing, it updates the task status flag stored in the buffer to provide a status basis for lock-free dequeue. The lock-free dequeue unit is used to work in conjunction with the single-track controller corresponding to the target buffer. The single-track controller detects the buffer status in real time. When the task status flag of the task to be played or paused is read, the lock-free dequeue unit realizes the lock-free reading of the audio task by independently operating the read pointer of the target buffer, and updates the task status flag after reading.

[0010] As an optional implementation, each circular buffer contains N ordered task storage bits, 1 read pointer (R), and 1 write pointer (W); each storage bit is bound to a unique task status identifier (S), and the initial state is 0; the task status identifier (S) is configured as: 0-idle, 1-waiting to play, 2-playing, 3-paused at breakpoint; The initial values ​​of the one read pointer (R) and one write pointer (W) are both 0, and their value range is locked from 0 to 19; N is a positive integer.

[0011] Therefore, this invention, by designing a lock-free concurrent sublayer with a circular buffer in the audio playback control module, creates an independent buffer for each audio track and implements lock-free enqueueing and dequeueing of audio tasks through independent read and write pointers, completely eliminating lock contention. At the same time, a lightweight fault-tolerant verification unit can be further configured to perform legality verification on the buffer at a set frequency (such as 10ms), automatically repairing problems such as pointer out-of-bounds and abnormal states, ensuring the continuous and stable operation of the system in a multi-threaded environment from the bottom layer, and solving the resource conflicts and concurrency stability problems caused by mutex locks and atomic operations in existing designs.

[0012] Meanwhile, through the parallel design of four core audio tracks (instances), it supports the simultaneous execution of multiple audio tasks (such as obstacle avoidance broadcasting and facial recognition prompts), meeting the parallel interaction needs of embodied intelligence in multiple scenarios. Combined with a configurable priority exclusive playback mechanism, high-priority tasks (such as faults and obstacle avoidance) can immediately interrupt low-priority tasks and record playback progress. After the high-priority task is completed, the low-priority task is automatically resumed, realizing task scheduling and breakpoint recovery under high and low priorities. Combined with the aforementioned lock-free concurrency mechanism, the submission and playback of multiple tasks are lock-free operations with no resource contention, significantly improving the efficiency of multi-task processing.

[0013] According to a second aspect of the present invention, a human-computer voice interaction method based on the aforementioned multimodal human-computer voice interaction system is also proposed, the method comprising the following steps: The system starts initialization, with the configuration management module, application adaptation module, voice interaction module, and audio playback control module starting initialization sequentially. Among them, after the audio controller of the audio playback control module completes single-instance initialization, it triggers the buffer management unit of the lock-free concurrent sub-unit of the circular buffer. According to the lock-free concurrent parameters provided by the configuration management module, it creates an independent circular buffer for each audio track and completes initialization. At the same time, the buffer is bound to the single audio track controller one by one. The single audio track controller loads the configuration parameters to complete the playback state initialization, and the audio playback control module enters the task receiving standby state. The configuration parsing unit reads the YAML configuration file and generates a global parameter dictionary after synchronous parsing. The application adaptation module, voice interaction module, and audio broadcast control module call the global parameter dictionary to complete the local parameter initialization. The application adaptation module and the voice interaction module generate corresponding standardized audio tasks based on the recognized valid scenarios and trigger events or commands, which are then submitted to the audio playback control module via ROS2 topics. The audio controller of the audio playback control module receives dual-channel standardized audio tasks, determines the target audio track of the task according to priority rules and audio track allocation rules, and then positions it to the circular buffer corresponding to the audio track as the target buffer. The lock-free enqueue unit of the audio playback control module receives the audio task assigned by the audio controller, independently operates the write pointer of the target buffer, writes the audio task into the target buffer storage bit and updates the task status flag of the storage bit to be played, thus completing the lock-free enqueue. The single-track controller bound to the target buffer detects the buffer status in real time. The lock-free dequeue unit independently operates the read pointer, reads the task with the status of "to be played" and passes it to the single-track controller. The single-track controller starts audio playback and simultaneously updates the task status to "playing", thus completing the lock-free dequeue and playback. After the audio task is completed, the single-track controller resets the status flag of the buffer storage bit to idle, and the buffer management unit realizes the cyclic reuse of the storage bit, waiting for new task input.

[0014] As an optional implementation, when the audio controller receives a preset high-priority task, it identifies all currently playing low-priority tasks and locates the circular buffer and storage bit corresponding to the low-priority task. The audio controller triggers the single-track controller to record the playback progress of low-priority tasks, and the lock-free concurrent unit of the circular buffer updates the status flag of the corresponding storage bit to breakpoint pause, thus completing the breakpoint marking. After a low-priority task is paused, a high-priority task is executed according to the process of unlocked enqueueing and unlocked dequeueing, and the single audio track controller controls the exclusive playback of the high-priority task. After the high-priority task finishes playing, the audio controller triggers the breakpoint recovery mechanism, locates the storage location of all breakpoint pauses, and the single-track controller, based on the recorded playback progress, rereads the task through the lockless dequeue unit, resumes playback from the breakpoint position, and updates the status flag to "playing," thus completing the breakpoint recovery of the low-priority task. It should be understood that all combinations of the foregoing concepts and the additional concepts described in more detail below may be considered part of the inventive subject matter of this disclosure, provided that such concepts do not contradict each other. Furthermore, all combinations of the claimed subject matter are considered part of the inventive subject matter of this disclosure.

[0015] The foregoing and other aspects, embodiments, and features of the teachings of the present invention will be more fully understood from the following description in conjunction with the accompanying drawings. Other additional aspects of the invention, such as features and / or beneficial effects of exemplary embodiments, will become apparent from the following description or may be learned through practice of specific embodiments according to the teachings of the present invention. Attached Figure Description

[0016] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component shown in the various figures may be denoted by the same reference numeral. For clarity, not every component is labeled in each figure. Embodiments of various aspects of the invention will now be described by way of example and with reference to the accompanying drawings.

[0017] Figure 1 This is a schematic diagram of a multimodal human-computer voice interaction system based on embodied intelligence according to an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram of a lock-free concurrent unit with a circular buffer according to an embodiment of the present invention.

[0019] Figure 3 This is a flowchart illustrating a human-computer voice interaction method according to an embodiment of the present invention. Detailed Implementation

[0020] To better understand the technical content of the present invention, specific embodiments are described below in conjunction with the accompanying drawings.

[0021] Various aspects of the invention are described in this disclosure with reference to the accompanying drawings, which illustrate numerous illustrative embodiments. The embodiments of this disclosure are not necessarily intended to encompass all aspects of the invention. It should be understood that the various concepts and embodiments described above, as well as those described in more detail below, can be implemented in any of many ways, because the concepts and embodiments disclosed herein are not limited to any particular implementation. Furthermore, some aspects of the invention disclosed may be used alone or in any suitable combination with other aspects of the invention disclosed.

[0022] {Example 1} Combined with appendix Figure 1-3 As shown, the multimodal human-computer voice interaction system based on embodied intelligence according to an embodiment of the present invention includes a voice interaction module, an audio playback control module, an application adaptation module, and a configuration management module.

[0023] The aforementioned voice interaction module is used for voice wake-up, voice recognition, semantic understanding and voice synthesis, and outputs voice interaction results and control commands to the audio playback control module.

[0024] The aforementioned application adaptation module has preset adaptation interfaces for multiple application scenarios. It identifies valid scenario trigger events based on robot sensor data and converts them into standardized audio tasks for output to the audio playback control module.

[0025] The aforementioned audio playback control module is used to determine the task priority and allocate audio tracks based on the standardized audio tasks submitted by the application adaptation module and the voice interaction module, locate the corresponding circular buffer of the audio track, and control the single audio track controller for playback and management by calling the lock-free enqueue / dequeue logic.

[0026] The aforementioned configuration management module parses and manages system parameters through YAML configuration files, including the management of audio parameters, voice interaction parameters, task priority parameters, and lock-free concurrency parameters.

[0027] The aforementioned audio playback control module is equipped with a ring buffer lock-free concurrent unit, which creates independent ring buffers for multiple audio tracks. Each ring buffer is bound to a single audio track controller, so that each audio track controller can only operate the read pointer and task of the corresponding buffer. The aforementioned audio playback control module completes the lock-free writing / reading of audio tasks based on the ring movement of the write / read pointer by scheduling the ring buffer lock-free concurrent unit, realizing the lock-free enqueueing and dequeueing of audio tasks scheduling control.

[0028] The write pointer is exclusively operated by the task submitter (the audio controller / unlocked queuing unit of the audio broadcast control module), while the read pointer is exclusively operated by the task executor (the single-track controller of the audio broadcast control module).

[0029] As an optional implementation, the aforementioned voice interaction module is the input source for human-computer voice interaction tasks, used to collect audio, recognize and synthesize TTS speech, and output voice interaction results and control commands to the audio playback control module, which specifically includes an audio processing unit and a voice interaction unit.

[0030] The aforementioned audio processing unit, developed based on C++ ROS2 nodes, is the fundamental audio processing unit for voice interaction. It is used to collect audio signals from the environment in real time through the service robot's microphone. For example, it collects raw audio signals from the environment in real time through the service robot's microphone device, providing raw data for speech recognition.

[0031] In another embodiment, the audio processing unit may also receive standardized audio output data from the audio playback control module to provide playback sources (such as explanatory audio, prompts, and TTS synthesized audio) for the service robot's speakers.

[0032] As an optional implementation, the audio processing unit can integrate an audio preprocessing module to perform noise reduction, filtering, sampling rate conversion, and other processing on the original audio signal to improve the audio signal quality and eliminate the interference of environmental noise on speech recognition. The preprocessed audio data is published in the form of ROS2 topics for the voice interaction unit to call.

[0033] The aforementioned voice interaction unit integrates a voice interaction client (aiui_v3_clientSDK) and a speech synthesis client (aiui_tts_clientSDK) to achieve intelligent processing of voice interaction. It completes voice wake-up, full-duplex voice interaction, semantic intent recognition, and TTS speech synthesis processing, converting the user's voice needs into standardized audio playback control commands or streaming audio tasks, which are then transmitted to the audio playback control module via ROS2 topics, achieving end-to-end collaboration between voice interaction and audio playback control.

[0034] In a specific example, the aforementioned voice interaction unit includes a voice wake-up subunit, a semantic recognition subunit, and a TTS speech synthesis subunit.

[0035] The voice wake-up subunit is used to monitor the audio data collected by the audio processing unit, detect the preset wake-up word (such as "Hello, Xiao X"), trigger the full-duplex voice interaction mode, and provide triggering conditions for subsequent semantic recognition.

[0036] The semantic recognition subunit is used to continuously receive audio data and perform speech recognition after wake-up. It parses the user's semantic intent through the handle_intent interface and distinguishes between audio playback control commands (such as playing the explanation of area A of the exhibition hall) and action control commands (such as moving forward and playing a prompt tone). The recognized commands are transmitted to the audio playback control module in ROS2 topic format.

[0037] For audio playback control commands, the command content and target audio identifier are directly transmitted to the AudioManager of the audio playback control module via ROS2 topics in a standardized format, whereby the latter handles subsequent task scheduling and playback control. For motion control linkage commands, on the one hand, the motion control command is transmitted to the service robot's motion control module to trigger the robot's corresponding action; on the other hand, a TTS synthesis request is sent to the TTS speech synthesis subunit to trigger the synthesis of the corresponding prompt sound, thus realizing the linkage between action and speech prompts.

[0038] The TTS speech synthesis subunit is used to generate TTS speech synthesis requests based on the action control commands or voice interaction requirements parsed by the semantic recognition subunit, such as "moving forward," "obstacle ahead, please avoid," and "arrived at exhibition hall B." It then obtains the synthesized streaming audio data through the speech synthesis client and encapsulates the synthesized streaming audio data into a system-standardized StreamTask streaming audio task. This task includes information such as task type, priority level (the priority is matched according to the trigger scenario by default, such as obstacle avoidance prompts being the highest priority), and audio data source, and is consistent with the task format of the application adaptation module before being transmitted to the audio playback control module for playback.

[0039] As an optional real-time approach, the aforementioned application adaptation module includes a scene docking unit, a sensor data processing unit, and a standardized task generation unit.

[0040] The scenario docking unit is used to provide adaptation interfaces for multiple preset application scenarios, and to configure task triggering rules, task types and priority levels for the scenarios.

[0041] In an embodiment of the present invention, the scenario docking unit integrates adaptation interfaces for eight preset typical industrial application scenarios, including: Obstacle avoidance broadcast scenario (connects to sensor data and triggers corresponding obstacle avoidance audio prompts); Face recognition prompt scenario (receive face recognition results and output identity confirmation prompt audio); Exhibition hall explanation scenario (play explanation audio according to preset route or triggered event); Customizable explanation scenarios (supports user-uploaded audio files and configuration of trigger conditions); Dance music scene (audio playback control synchronized with robot dance movements); Real-time voice interaction scenarios (integrating with AIUI to achieve human-computer dialogue and command response); Equipment fault notification scenarios (receiving equipment status signals and outputting fault alarm audio); and Operation guidance scenario (output step-by-step audio guidance according to the operation process).

[0042] It should be understood that the eight typical industrial application scenarios in the examples above are provided as examples in a non-exhaustive manner. In other implementations, more or fewer, or other application scenarios, may be configured.

[0043] The sensor data processing unit receives real-time data collected by the service robot's sensors and identifies valid scene trigger events. These sensors, especially various hardware sensors that indicate the service robot's status, include, but are not limited to, obstacle avoidance, infrared, facial recognition, and equipment status sensors. Based on the real-time data, the unit performs preprocessing on the sensor signals, such as noise reduction and filtering, to identify valid scene trigger events, such as an obstacle avoidance sensor detecting an obstacle or an infrared sensor detecting a visitor entering the robot.

[0044] The standardized task generation unit is used to convert valid trigger events identified by the sensor data processing unit into standardized audio tasks. The task includes task type, priority level, audio data source and target audio track parameters, providing a unified task input for the audio playback control module. The aforementioned task types include FileTask (local file) and StreamTask (streaming audio).

[0045] As an optional implementation, the aforementioned configuration management module provides system-wide parameter management and provides unified parameter configuration and dynamic update support for other modules. It includes a configuration parsing unit, a parameter caching unit, and a timed polling unit.

[0046] The configuration parsing unit reads the YAML configuration file and parses audio parameters, voice interaction parameters, task priority parameters, and lock-free concurrency parameters. The aforementioned lock-free concurrency parameters include buffer capacity, pointer rules, and verification frequency. After parsing, a standardized global parameter dictionary is generated. Simultaneously, this configuration parsing unit allows the configuration layer to perform validity checks on the parsed parameters according to preset rules; invalid parameters automatically trigger default value restoration.

[0047] The parameter caching unit stores the global parameter dictionary generated by the configuration parsing unit, provides a real-time parameter calling interface and an intermediate carrier for dynamic parameter updates, receives new parameters from the timed polling unit and completes cache updates to ensure parameter consistency across modules.

[0048] The timed polling unit is used to detect the modification status of the YAML configuration file at a preset frequency. If a file update is detected, the configuration parsing unit is immediately triggered to re-parse the parameters and synchronize the new parameters to the parameter cache unit to achieve dynamic parameter updates.

[0049] As an optional approach, operations and maintenance personnel can modify the YAML configuration file (which can modify original system parameters or lock-free concurrency parameters) through the configuration interface of the application adaptation module. Simultaneously, after the configuration management module's periodic polling unit detects file modification, it will trigger the configuration parsing unit to re-parse the parameters and update the cached data in the parameter caching unit, achieving dynamic updates and consistency of global parameters.

[0050] • The configuration management module publishes parameter update notifications to all modules via ROS2 topics → Each module and its internal sub-modules load the new parameters from the configuration management module in real time → Local parameter updates are completed. The entire process does not require a system restart and does not affect the normal operation of the service robot.

[0051] As an optional implementation, the aforementioned audio playback control module includes an audio controller (AudioManager), a single-track controller (TrackAudioController), a standardized encapsulation component (AudioTask), and a ring buffer lock-free concurrent unit to realize lock-free scheduling of audio tasks, parallel playback of multiple audio tracks, priority exclusive playback, and breakpoint recovery control.

[0052] The aforementioned audio controller operates in singleton mode and is the core of global scheduling. It receives standardized audio tasks submitted by the application adaptation module and the voice interaction module, performs task priority judgment and audio track allocation, and locates the corresponding circular buffer as the target buffer. It controls the single audio track controller corresponding to the target buffer for playback and management by calling the lock-free enqueue / dequeue logic.

[0053] The aforementioned single-track controller is configured with multiple instances according to the robot scenario, each bound to a corresponding physical track. One instance corresponds to one track / physical channel. The single-track controller receives scheduling instructions from the audio controller to complete the playback, pause, stop, volume adjustment, and playback status management of the single track, and records the playback progress of the audio task to support breakpoint recovery.

[0054] The aforementioned standardized encapsulation components are used for the standardized encapsulation of audio tasks, respectively implementing the definition of basic audio task interfaces, the encapsulation of local audio file playback tasks, and the encapsulation of streaming audio playback tasks; under them are the AudioTask base class, the FileTask file audio task class, and the StreamTask streaming audio task class, for unified processing of different types of audio tasks.

[0055] The aforementioned lock-free concurrent unit of the circular buffer is used to independently operate the write pointer of the target buffer according to the circular buffer located by the audio controller, and to cooperate with the aforementioned single-track controller to independently operate the read pointer of the target buffer, so as to realize the lock-free enqueueing, lock-free dequeueing and playback of audio tasks.

[0056] As an optional implementation method, combined with Figure 2 As shown, the aforementioned lock-free concurrent unit of the circular buffer includes a buffer management unit, a lock-free enqueue unit, and a lock-free dequeue unit; The buffer management unit is used to create independent circular buffers for multiple audio tracks. In this example, four tracks are used, such as welcome sound, explanation sound, obstacle avoidance sound, and fault sound. Each circular buffer is bound to a single audio track controller, so that each audio track controller can only operate the read pointer and task of the corresponding buffer, ensuring the isolated processing of each audio track task.

[0057] The lock-free enqueue unit is used to receive audio tasks assigned by the audio controller. It realizes lock-free writing of audio tasks by independently operating the write pointer of the target buffer. After writing, it updates the task status flag stored in the buffer to provide a status basis for lock-free dequeueing and eliminate lock contention.

[0058] The lock-free dequeue unit is used to work in conjunction with the single-track controller corresponding to the target buffer. The single-track controller detects the buffer status in real time. When the task status flag of a task to be played or paused is read, the lock-free dequeue unit realizes the lock-free reading of the audio task by independently operating the read pointer of the target buffer. After reading, the task status flag is updated to ensure that there is no resource contention between the playback thread and the submission thread.

[0059] As an optional implementation, each circular buffer contains N ordered task storage bits, 1 read pointer (R), and 1 write pointer (W); each storage bit is bound to a unique task status identifier (S), and the initial state is 0; the task status identifier (S) is configured as: 0-idle, 1-waiting to play, 2-playing, 3-paused at breakpoint.

[0060] The initial values ​​of the aforementioned read pointer (R) and write pointer (W) are both 0. The value range of the pointers corresponds to the task storage bit N, and the value range is from 0 to N-1, where N is a positive integer. In this example, we take N=20 as an example for explanation, and the value range of the read pointer (R) and write pointer (W) is locked to 0~19.

[0061] In other embodiments, N can be dynamically adjusted according to system memory requirements.

[0062] Therefore, during system initialization and configuration loading, the circular buffer is initialized. Based on the parsed parameters from the configuration management module, the audio playback control module reads the lock-free concurrency parameters from the global parameter dictionary, creating four independent circular buffer instances for the four audio tracks. Each buffer is initialized, generating 20 ordered task storage bits, each bound to a unique task status identifier (S), with an initial state of 0 (idle). Simultaneously, the read pointer (R) and write pointer (W) for each buffer are initialized, with initial values ​​of 0 and a locked value range of 0~19. The four circular buffers are then bound one-to-one to four single-track controllers, ensuring that each track controller can only operate on the read pointer and task of its corresponding buffer, preventing cross-track interference.

[0063] As an optional implementation, the lock-free concurrent unit of the circular buffer is also equipped with a fault-tolerant verification unit, which is used to perform legality verification on the circular buffer at a preset frequency, including verification of the read / write pointer value range, task status identifier, and the validity of the storage bit task.

[0064] For example, when an anomaly is detected (pointer out of bounds, state anomaly, empty task), the pointer is automatically reset and the state is restored, and the anomaly log is recorded. The log is then reported to the service robot's monitoring terminal via ROS2 topic to improve system stability.

[0065] As an optional implementation, the aforementioned audio controller presets an audio task priority level. When a high-priority task enters the task queue, it controls the exclusive playback scheduling of the high-priority task and records the playback progress of the low-priority task. The aforementioned audio controller triggers an automatic recovery mechanism after the high-priority task is completed, instructing each single-track controller to resume playback of the corresponding low-priority task according to the recorded playback progress, thereby achieving breakpoint recovery control for low-priority tasks.

[0066] Combined with appendix Figure 3 The embodied intelligence multimodal human-computer voice interaction system described in the above embodiments, and the human-computer voice interaction method implemented therein, include the following steps: Step S101: System startup initialization. The configuration management module, application adaptation module, voice interaction module, and audio broadcast control module start initialization in sequence. After the configuration management module completes initialization, the application adaptation module, voice interaction module, and audio broadcast control module start. Each internal sub-module / functional unit completes its own initialization and sends a parameter call request to the parameter cache unit of the configuration management module to establish the basic link for parameter call. After the AudioManager of the audio playback control module completes singleton initialization, it triggers the buffer management unit of the lock-free concurrent subunit of the circular buffer. According to the lock-free concurrent parameters provided by the configuration management module, it creates an independent circular buffer for each audio track and completes initialization. At the same time, the buffer is bound to the single audio track controller one by one. The single audio track controller loads the configuration parameters to complete the playback state initialization, and the audio playback control module enters the task receiving standby state. Step S102: The aforementioned configuration parsing unit reads the YAML configuration file and generates a global parameter dictionary after synchronous parsing. In particular, the global parameter dictionary is generated after legality verification. Then, the application adaptation module, voice interaction module, and audio broadcast control module call the global parameter dictionary to complete the local parameter initialization. In step S103, the application adaptation module and the voice interaction module generate corresponding standardized audio tasks based on the recognized valid scene trigger events or instructions, and submit them to the audio playback control module through ROS2 topics. Step S104: The audio controller of the audio playback control module receives dual-channel standardized audio tasks, determines the target audio track of the task according to the priority rules and audio track allocation rules, and then positions it to the circular buffer corresponding to the audio track as the target buffer. Step S105: The lock-free enqueue unit of the audio playback control module receives the audio task assigned by the audio controller, independently operates the write pointer of the target buffer, writes the audio task into the target buffer storage bit and updates the task status flag of the storage bit to be played, thus completing the lock-free enqueue. Step S106: The single-track controller bound to the target buffer detects the buffer status in real time. The lockless dequeue unit independently operates the read pointer, reads the task with the status of waiting to be played and passes it to the single-track controller. The single-track controller starts audio playback and updates the task status to play at the same time, thus completing the lockless dequeue and playback. Step S107: After the audio task is completed, the single-track controller resets the status flag of the buffer storage bit to idle, and the buffer management unit realizes the cyclic reuse of the storage bit, waiting for new task input.

[0067] The buffer management unit monitors the task status of each buffer storage bit in real time. When the task in a storage bit is completed and marked as idle, it is included in the reusable queue to realize the recycling of buffer storage resources.

[0068] In step S103, the internal sub-modules of the application adaptation module and the voice interaction module are processed in a pipeline manner and submitted to the AudioManager of the audio playback control module in the form of standardized audio tasks through ROS2 topics, forming dual-channel task input.

[0069] As described in the aforementioned embodiments, the specific process for generating standardized audio tasks is as follows: (1) Audio task generation using the application adaptation module: The sensor data processing unit receives real-time data from the service robot's sensors, preprocesses it, identifies valid triggering events, and transmits them to the scene docking unit. The scene docking unit matches task attributes according to scene binding rules and transmits them to the standardized task generation unit. The standardized task generation unit converts the data into a standardized audio task and submits it to the AudioManager via ROS2 topic. (2) Audio task generation for the voice interaction module: The audio acquisition subunit of the audio processing unit acquires the raw audio, which is then processed by the audio preprocessing subunit for noise reduction and filtering before being published to the voice interaction unit. The voice wake-up subunit of the voice interaction unit detects the wake-up word and triggers the semantic recognition subunit to parse the intent: if it is a play control command, it is directly transmitted to the AudioManager; if it is an action control command, it triggers the TTS speech synthesis subunit to generate streaming audio data, which is then packaged into a standardized audio task and submitted to the AudioManager.

[0070] In conjunction with the priority scheduling processing of the foregoing embodiments, in the embodiments of the present invention, when the audio controller receives a preset high-priority task, it identifies all currently playing low-priority tasks and locates the circular buffer and storage bit corresponding to the low-priority task. The audio controller triggers the single-track controller to record the playback progress of low-priority tasks, and the lock-free concurrent unit of the circular buffer updates the status flag of the corresponding storage bit to breakpoint pause, thus completing the breakpoint marking. After a low-priority task is paused, a high-priority task is executed according to the process of unlocked enqueueing and unlocked dequeueing, and the single audio track controller controls the exclusive playback of the high-priority task. After a high-priority task finishes playing, the audio controller triggers the breakpoint recovery mechanism, locates the storage location where all breakpoints were paused, and the single-track controller rereads the task through the unlocked dequeue unit according to the recorded playback progress, continues playback from the breakpoint position, and updates the status flag to playing, thus completing the breakpoint recovery of the low-priority task.

[0071] This completes the processing flow of the multimodal human-computer voice interaction method.

[0072] While the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A multi-modal human-machine speech interaction system based on embodied intelligence, characterized in that, It includes a voice interaction module, an audio playback control module, an application adaptation module, and a configuration management module; The voice interaction module is used for voice wake-up, voice recognition, semantic understanding and voice synthesis, and outputs voice interaction results and control commands to the audio playback control module; The application adaptation module presets adaptation interfaces for multiple application scenarios, identifies effective scenario trigger events based on robot sensor data, and converts them into standardized audio tasks for output to the audio playback control module. The audio playback control module is used to determine the task priority and allocate audio tracks according to the standardized audio tasks submitted by the application adaptation module and the voice interaction module, locate the circular buffer corresponding to the audio track, and control the single audio track controller for playback and management by calling the lock-free enqueue / dequeue logic. The configuration management module parses and manages system parameters through YAML configuration files, including the management of audio parameters, voice interaction parameters, task priority parameters, and lock-free concurrency parameters. The audio playback control module is equipped with a ring buffer lock-free concurrent unit, which creates independent ring buffers for multiple audio tracks. Each ring buffer is bound to a single audio track controller, so that each audio track controller can only operate the read pointer and task of the corresponding buffer. The audio playback control module completes the lock-free writing / reading of audio tasks based on the ring movement of the write / read pointer by scheduling the ring buffer lock-free concurrent unit, realizing the lock-free enqueueing and dequeueing of audio tasks scheduling control.

2. The multimodal human-computer voice interaction system based on embodied intelligence according to claim 1, characterized in that, The voice interaction module includes an audio processing unit and a voice interaction unit; The audio processing unit is used to collect audio signals from the environment in real time through the service robot's microphone; The voice interaction unit integrates a voice interaction client and a voice synthesis client to realize voice wake-up, full-duplex voice interaction, semantic intent recognition, and TTS voice synthesis processing.

3. The multimodal human-computer voice interaction system based on embodied intelligence according to claim 2, characterized in that, The voice interaction unit includes a voice wake-up subunit, a semantic recognition subunit, and a TTS speech synthesis subunit. The voice wake-up subunit is used to monitor the audio data collected by the audio processing unit and detect the preset wake-up word to trigger the full-duplex voice interaction mode. The semantic recognition subunit is used to continuously receive audio data and perform speech recognition after wake-up. It parses the user's semantic intent through the handle_intent interface, distinguishes between audio playback control commands and action control commands, and transmits the recognized commands to the audio playback control module in ROS2 topic format. The TTS speech synthesis subunit is used to generate TTS speech synthesis requests based on the action control commands or voice interaction requirements parsed by the semantic recognition subunit. It then obtains the synthesized streaming audio data through the speech synthesis client, encapsulates it into a StreamTask standardized audio task, and transmits it to the audio playback control module for playback.

4. The multimodal human-computer voice interaction system based on embodied intelligence according to claim 1, characterized in that, The application adaptation module includes: The scenario docking unit is used to provide adaptation interfaces for multiple preset application scenarios, and to configure task triggering rules, task types and priority levels for the scenarios; The sensor data processing unit is used to receive real-time data collected by the service robot's sensors and identify valid scene trigger events; The standardized task generation unit is used to convert valid trigger events identified by the sensor data processing unit into standardized audio tasks. The tasks include task type, priority level, audio data source, and target audio track parameters, providing a unified task input for the audio playback control module. The task types include FileTask (local file) and StreamTask (streaming audio).

5. The multimodal human-computer voice interaction system based on embodied intelligence according to claim 1, characterized in that, The configuration management module includes a configuration parsing unit, a parameter caching unit, and a timed polling unit; The configuration parsing unit is used to read the YAML configuration file and parse audio parameters, voice interaction parameters, task priority parameters and lock-free concurrency parameters. The lock-free concurrency parameters include buffer capacity, pointer rules and verification frequency. After parsing, a standardized global parameter dictionary is generated. The parameter caching unit is used to store the global parameter dictionary generated by the configuration parsing unit, provides a real-time parameter calling interface and an intermediate carrier for dynamic parameter updates, receives new parameters from the timed polling unit and completes cache updates; The timed polling unit is used to detect the modification status of the YAML configuration file at a preset frequency. If a file update is detected, the configuration parsing unit is immediately triggered to re-parse the parameters and synchronize the new parameters to the parameter cache unit to achieve dynamic parameter updates.

6. The multimodal human-computer voice interaction system based on embodied intelligence according to any one of claims 1-5, characterized in that, The audio playback control module includes an audio controller, a single-track controller, standardized encapsulation components, and a ring buffer lock-free concurrent unit; The audio controller operates in singleton mode and is used to receive standardized audio tasks submitted by the application adaptation module and the voice interaction module. It performs task priority judgment, audio track allocation, locates the circular buffer corresponding to the audio track and uses it as the target buffer. It controls the single audio track controller corresponding to the target buffer for playback and management by calling the lock-free enqueue / dequeue logic. The single-track controller is configured with multiple instances according to the robot scenario, and each instance is bound to a corresponding physical track. Each instance corresponds to one track / physical channel. The single-track controller receives scheduling instructions from the audio controller to complete the playback, pause, stop, volume adjustment and playback status management of the single track, and records the playback progress of the audio task to provide support for breakpoint recovery. The standardized encapsulation component is used for the standardized encapsulation of audio tasks, and respectively implements the definition of basic audio task interfaces, the encapsulation of local audio file playback tasks, and the encapsulation of streaming audio playback tasks. The lock-free concurrent unit of the circular buffer is used to independently operate the write pointer of the target buffer according to the circular buffer located by the audio controller, and to cooperate with the single track controller to independently operate the read pointer of the target buffer, so as to realize the lock-free enqueueing, lock-free dequeueing and playback of audio tasks.

7. The multimodal human-computer voice interaction system based on embodied intelligence according to claim 6, characterized in that, The audio controller presets an audio task priority hierarchy. When a high-priority task enters the task queue, it controls the exclusive playback scheduling of the high-priority task and records the playback progress of the low-priority task. The audio controller triggers an automatic recovery mechanism after a high-priority task is completed, instructing each single-track controller to resume playback of the corresponding low-priority task according to the recorded playback progress, thereby achieving breakpoint recovery control for low-priority tasks.

8. The multimodal human-computer voice interaction system based on embodied intelligence according to claim 6, characterized in that, The lock-free concurrent unit of the circular buffer includes a buffer management unit, a lock-free enqueue unit, and a lock-free dequeue unit; The buffer management unit is used to create independent circular buffers for multiple audio tracks. Each circular buffer is bound to a single audio track controller, so that each audio track controller can only operate the read pointer and task of the corresponding buffer. The lock-free enqueue unit is used to receive audio tasks assigned by the audio controller. It realizes lock-free writing of audio tasks by independently operating the write pointer of the target buffer. After writing, it updates the task status flag stored in the buffer to provide a status basis for lock-free dequeue. The lock-free dequeue unit is used to work in conjunction with the single-track controller corresponding to the target buffer. The single-track controller detects the buffer status in real time. When the task status flag of the task to be played or paused is read, the lock-free dequeue unit realizes the lock-free reading of the audio task by independently operating the read pointer of the target buffer, and updates the task status flag after reading.

9. The multimodal human-computer voice interaction system based on embodied intelligence according to claim 8, characterized in that, Each circular buffer contains N ordered task storage bits, 1 read pointer (R), and 1 write pointer (W); each storage bit is bound to a unique task status identifier (S), with an initial state of 0; the task status identifier (S) is configured as: 0-idle, 1-waiting to play, 2-playing, 3-paused at breakpoint; The initial values ​​of the one read pointer (R) and one write pointer (W) are both 0, and their value range is locked from 0 to 19; N is a positive integer.

10. The human-computer voice interaction method implemented by the multimodal human-computer voice interaction system based on embodied intelligence according to any one of claims 1-9, characterized in that, The method includes the following steps: The system starts and initializes, with the configuration management module, application adaptation module, voice interaction module, and audio broadcast control module starting and initializing in sequence. The configuration parsing unit reads the YAML configuration file and generates a global parameter dictionary after synchronous parsing. The application adaptation module, voice interaction module, and audio broadcast control module call the global parameter dictionary to complete the local parameter initialization. The application adaptation module and the voice interaction module generate corresponding standardized audio tasks based on the recognized valid scenarios and trigger events or commands, which are then submitted to the audio playback control module via ROS2 topics. The audio controller of the audio playback control module receives dual-channel standardized audio tasks, determines the target audio track of the task according to priority rules and audio track allocation rules, and then positions it to the circular buffer corresponding to the audio track as the target buffer. The lock-free enqueue unit of the audio playback control module receives the audio task assigned by the audio controller, independently operates the write pointer of the target buffer, writes the audio task into the target buffer storage bit and updates the task status flag of the storage bit to be played, thus completing the lock-free enqueue. The single-track controller bound to the target buffer detects the buffer status in real time. The lock-free dequeue unit independently operates the read pointer, reads the task with the status of "to be played" and passes it to the single-track controller. The single-track controller starts audio playback and simultaneously updates the task status to "playing", thus completing the lock-free dequeue and playback. After the audio task is completed, the single-track controller resets the status flag of the buffer storage bit to idle, and the buffer management unit realizes the cyclic reuse of the storage bit, waiting for new task input.

11. The human-computer voice interaction method according to claim 10, characterized in that, When the audio controller receives a preset high-priority task, it identifies all currently playing low-priority tasks and locates the circular buffer and storage bit corresponding to the low-priority task. The audio controller triggers the single-track controller to record the playback progress of low-priority tasks, and the lock-free concurrent unit of the circular buffer updates the status flag of the corresponding storage bit to breakpoint pause, thus completing the breakpoint marking. After a low-priority task is paused, a high-priority task is executed according to the process of unlocked enqueueing and unlocked dequeueing, and the single audio track controller controls the exclusive playback of the high-priority task. After a high-priority task finishes playing, the audio controller triggers the breakpoint recovery mechanism, locates the storage location where all breakpoints were paused, and the single-track controller rereads the task through the unlocked dequeue unit according to the recorded playback progress, continues playback from the breakpoint position, and updates the status flag to playing, thus completing the breakpoint recovery of the low-priority task.

Images