A multimodal voice interaction method and device

Abstract

This invention discloses a multimodal voice interaction method and apparatus, relating to the field of robot control technology. By constructing a task orchestration mechanism centered on task requirements, it enables various capability modules to participate in task execution around the same task objective, avoiding problems such as task execution fragmentation and conflicts caused by independent execution of each capability module. Simultaneously, it reduces issues such as repeated calls, repeated interactions, and repeated feedback during the execution of complex multimodal voice tasks, thereby improving the continuity and reliability of complex multimodal voice task execution. By establishing a state management approach that spans the entire process of task creation, execution, completion, and feedback for complex multimodal voice tasks, the task state can be continuously updated as the task execution progresses, avoiding problems such as repeated execution, erroneous execution, and inconsistencies between feedback content and the actual execution state due to delayed task state.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a multimodal voice interaction method and device for intelligent robots. Background Technology

[0002] With the increasing application of intelligent robots in scenarios such as companionship, inspection, and guide services, voice interaction has become an important way for users to control robots and obtain their services. Existing intelligent robots are typically equipped with microphones, speakers, cameras, navigation and positioning modules, and motion control modules, enabling functions such as voice wake-up, intent recognition, voice broadcasting, control, navigation and positioning, and environmental perception. In current technologies, robot voice interaction is usually implemented using cloud-based speech recognition models or large-scale intelligent dialogue models. After the robot collects the user's voice, it sends the audio data to the cloud, where speech recognition, semantic understanding, and response generation are performed. The processed results are then returned to the robot for voice broadcasting or execution of corresponding operations. When the user's voice involves tasks such as motion control, navigation services, or environmental perception, the robot typically further invokes the motion control module, navigation planning module, camera image acquisition module, and third-party service interfaces to complete the corresponding task.

[0003] However, as robot application scenarios become increasingly complex, user voice requests have evolved from simple question-and-answer sessions to complex tasks involving multiple capabilities such as navigation, environmental observation, information retrieval, and device control. For example, users might request tasks like "Take me to the nearest pharmacy," "See what's ahead," "Take a picture and send it to my family," or "Check the weather for my current location." These tasks not only require speech recognition and semantic understanding but also necessitate combining the robot's current location, operational status, and external service information, and invoking different functional modules such as navigation, environmental perception, motion control, and information retrieval to collaboratively complete the task. Especially in multi-turn voice interactions, subsequent user voice requests often depend on the results of previous interactions. For example, after requesting "Find a nearby pharmacy," a user might request "Navigate to the first location," or after requesting "Check the environment ahead," a user might request "Take a picture and send it to my family." Such interconnected tasks require utilizing historical interaction results and the current robot operational status to generate subsequent execution strategies, thus necessitating continuous management of the task and execution states throughout the conversation. In existing technologies, functional modules typically operate as independent services or functional units, each completing only its own functional processing, lacking a unified state management and task collaborative execution mechanism for complex voice tasks. When a user request requires the participation of multiple functional modules, the system struggles to uniformly manage the task lifecycle, module call order, execution status, and feedback results, easily leading to problems such as state mismatch, execution conflicts, duplicate responses, and inconsistent feedback during task execution. For example, in voice navigation scenarios, the destination proposed by the user is often not a pre-configured fixed navigation point, but rather a dynamic target such as "the nearest toilet" or "nearby bus stop." Such targets require obtaining candidate location information based on the robot's current location, and generating a navigation task that the robot can execute after user confirmation.

[0004] Existing voice interaction systems struggle to achieve a stable transition from natural language destination descriptions to robot navigation tasks. In voice environment understanding scenarios, the robot needs to acquire and analyze the current environment image based on the user's voice requests. When the user repeatedly initiates environment viewing requests, historical interaction information and historical environment information may influence the current analysis results, thereby reducing the correspondence and accuracy between the environment understanding results and the current scene. Furthermore, in voice control scenarios, after the robot completes its local control task, the cloud system may continue to return responses, resulting in repetitive broadcasts, delayed feedback, and inconsistencies between the feedback content and the actual execution status. Simultaneously, the robot's own broadcast speech may be re-acquired and used in speech recognition, affecting the stability and accuracy of subsequent voice interaction processes.

[0005] Therefore, how to provide a multimodal voice interaction method and system, and how to establish a unified state management and task collaborative execution mechanism to achieve coordinated control of the understanding, organization, execution and feedback process of complex voice tasks, thereby improving the interaction accuracy, execution reliability and task processing consistency of robots in complex interaction scenarios, is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of the above problems, the present invention provides a multimodal voice interaction method and apparatus for overcoming or at least partially solving the above problems.

[0007] This invention provides the following solution: A multimodal voice interaction method, comprising: Audio data in pulse code modulation format is collected through a microphone. When the robot is in a wake-up state, the audio data is preprocessed and then uploaded to the cloud. The preprocessed audio data is converted into text information, and semantic analysis is performed on the text information to determine the task type, extract the target object, task parameters and contextual information, and generate structured semantic results. Based on the structured semantic results, task requirements are generated, including task identifier, task type, task objective, task parameters, task status, and capability requirement information; state management information associated with the session is established, including session status, task status, capability invocation status, and execution feedback status; for tasks involving environmental awareness, a unique task identifier is generated for each task, and the collected environmental data is bound and stored with the current task identifier. A task execution strategy is generated based on the task requirements and the status management information. When the capability requirement information involves dynamic target location query, the information query module is invoked to query a set of candidate locations based on the current location information and the target category. The set of candidate locations is then fed back to the user for confirmation. Based on the user's confirmation, the final navigation target location is determined, and a navigation task requirement is generated. When the capability requirement information involves multiple capabilities, a capability execution chain is established. The calling order is determined according to the data dependencies between capabilities, and the robot capability module is invoked to execute the task. During execution, only environmental data bound to the current task identifier is called for analysis and processing. During task execution, the robot continuously receives execution results returned by the robot capability module and updates the task status in real time based on the execution results. When a voice request that depends on historical interaction results is received, the robot completes, updates, or constrains the current task requirements based on the historical task status saved in the session status. When there are abnormalities in environmental analysis, path failure, target unreachability, positioning abnormalities, navigation timeouts, query failures, query timeouts, or external service abnormalities, the robot determines that the current capability call has failed, immediately stops the subsequent execution process of the current task, updates the corresponding task status to terminated, and records the corresponding abnormal reason. The execution results are filtered and fused to generate feedback text. The text-to-speech module is used to synthesize speech and broadcast it. During the broadcast, the microphone is controlled to stop collecting or pause audio data uploading through a message communication mechanism. After the broadcast is completed, voice input processing is resumed and the current feedback status is updated.

[0008] Preferably, the preprocessing includes resampling and converting the audio data into a standard speech data format with a preset sampling rate; the cloud includes a cloud-based language model for speech recognition and a cloud-based visual model for image analysis.

[0009] Preferably, the task status includes created, pending execution, executing, completed, and terminated; when a stop command is received, the task control center updates the task status of the corresponding running task to terminated to prevent the current task from continuing to execute.

[0010] Preferably: Invoking the environment awareness module includes: Image data acquired by visible light cameras and depth cameras is obtained and associated with the unique task identifier. In multiple consecutively initiated environment viewing requests, the corresponding environment data is distinguished based on different task identifiers, and only the image data associated with the current task identifier is input into the visual big model for analysis.

[0011] Preferably, invoking the positioning and navigation module specifically includes: Obtain positioning information including latitude and longitude, positioning timestamp, positioning status, and positioning accuracy; stop navigation processing when the positioning status is abnormal or the positioning accuracy is lower than a preset threshold; for scenarios involving generalized navigation requests, supplement the current task requirements with the robot's current valid positioning information as query conditions.

[0012] Preferably, invoking the motion control module specifically includes: The robot receives motion control requests directly from users or navigation paths generated by the positioning and navigation module; it dynamically generates motion control commands based on the control parameters in the motion control request or the navigation path, and sends them to the motor driver through the robot's underlying control interface; during movement, it adjusts the robot's motion state in real time based on obstacle information obtained by lidar and ultrasonic ranging sensors.

[0013] Preferably, invoking the information query module specifically includes: Based on the target category information in the task execution strategy and the current location information, perform location retrieval processing; generate a set of candidate locations containing destination names and corresponding latitude and longitude information; and write the query results into the current task requirements and historical session information for subsequent task calls.

[0014] Preferably, filtering the execution results includes: Identify the timestamp of the execution result and the associated task identifier; remove expired feedback information whose timestamp is earlier than the current task status update time; filter execution results that do not match the current task status and duplicate execution results generated by module calls.

[0015] Preferably, the generated feedback text includes: For single-capability tasks, the feedback content is generated directly from the execution result returned by the corresponding capability; for multi-capability collaborative tasks, a unified feedback content is generated based on the correlation between the results returned by each capability; during the execution of a navigation task, the navigation execution status returned by the positioning and navigation module is fused with the environmental description information returned by the environmental perception module to generate the feedback content.

[0016] A multimodal voice interaction device for executing the above-described multimodal voice interaction method, the device comprising: The speech recognition and conversation management unit is used to collect audio data in pulse code modulation format through a microphone and upload the preprocessed audio data to the cloud when the robot is in a wake-up state. The speech content understanding and intent parsing unit is used to convert the uploaded audio data into text information, perform semantic analysis on the text information, determine the task type, extract the target object, task parameters and contextual information, and generate structured semantic results. The task requirement generation and status management unit is used to generate task requirements based on the structured semantic results. The task requirements include task identifier, task type, task objective, task parameters, task status, and capability requirement information. It also establishes status management information associated with the session, including session status, task status, capability call status, and execution feedback status. For tasks involving environmental awareness, it generates a unique task identifier for each task and binds and stores the collected environmental data with the current task identifier. The task orchestration and capability invocation unit is used to generate a task execution strategy based on the task requirements and the status management information. When the capability requirement information involves dynamic target location query, the information query module is invoked to query a set of candidate locations based on the current positioning information and the target category. The set of candidate locations is then fed back to the user for confirmation. Based on the user's confirmation, the final navigation target location is determined and a navigation task requirement is generated. When the capability requirement information involves multiple capabilities, a capability execution chain is established. The invocation order is determined according to the data dependencies between capabilities, and the robot capability module is invoked to execute the task. During execution, only environmental data bound to the current task identifier is invoked for analysis and processing. The task execution and status update unit is used to continuously receive the execution results returned by the robot capability module during task execution, and update the task status in real time according to the execution results; when a voice request that depends on the historical interaction results is received, the current task requirements are completed, updated or constrained according to the historical task status saved in the session status; when the environment analysis is abnormal, the path fails, the target is unreachable, the positioning is abnormal, the navigation times out, or the query fails, the query times out, or the external service is abnormal, the current capability call is determined to have failed, the subsequent execution process of the current task is stopped immediately, the corresponding task status is updated to terminated, and the corresponding abnormal reason is recorded. The execution result feedback and voice broadcasting unit is used to filter and fuse the execution results, generate feedback text, synthesize speech using a text-to-speech module, and broadcast it. During the broadcasting process, the microphone is controlled to stop collecting or pause audio data uploading through a message communication mechanism. After the broadcasting is completed, voice input processing is resumed, and the current feedback status is updated.

[0017] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: This application provides a multimodal voice interaction method and apparatus. By constructing a task orchestration mechanism centered on task requirements, each capability module participates in task execution around the same task objective, avoiding problems such as task execution fragmentation and execution conflicts caused by independent execution of each capability module. At the same time, it reduces problems such as repeated calls, repeated interactions, and repeated feedback generated during the execution of complex multimodal voice tasks, thereby improving the execution continuity and reliability of complex multimodal voice tasks.

[0018] By establishing a state management approach that spans the entire process of task creation, execution, termination, and feedback for complex multimodal speech tasks, the task state can be continuously updated as the task is executed. This avoids problems such as duplicate execution, erroneous execution, and inconsistencies between feedback content and actual execution state caused by lag in task state, thereby improving the execution consistency and interaction reliability of complex multimodal speech interaction processes.

[0019] By constructing a mechanism for generating task requirements, orchestrating tasks, and coordinating the execution of capabilities, the intentions expressed by users during natural language interaction are gradually converted into task semantics that the robot can directly execute, and further converted into corresponding robot actions. This achieves an effective connection between natural language interaction and actual robot actions, enabling users to directly drive the robot to complete complex tasks using natural language, thereby improving the intelligence level of complex multimodal voice interaction.

[0020] By uniformly processing, filtering, and integrating the feedback results of each capability module during robot execution, conflicts and duplicate feedback between execution results from different sources are eliminated. At the same time, feedback information that has become invalid or is inconsistent with the current task status is removed. This avoids problems such as repeated broadcasts, erroneous broadcasts, and delayed feedback in complex multimodal voice interaction, thereby improving the accuracy and consistency of robot execution result feedback. Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0022] Figure 1 This is a flowchart of a multimodal voice interaction method provided in an embodiment of the present invention; Figure 2 This is a system overall architecture diagram provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of a multimodal voice interaction device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a multimodal voice interaction device provided in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0024] See Figure 1 This invention provides a multimodal voice interaction method, such as... Figure 1 As shown, the method may include: S101: Acquire audio data in pulse code modulation format via microphone. When the robot is in a wake-up state, preprocess the audio data and upload it to the cloud. In specific implementation, the preprocessing of this invention may include resampling processing and converting the audio data into a standard speech data format with a preset sampling rate. The cloud includes a cloud-based language model for speech recognition and a cloud-based visual model for image analysis.

[0025] In practice, the robot collects user voice data via an external microphone. The collected audio data is stored and transmitted in PCM (Pulse Code Modulation) format, and the input sampling rate can be set according to the device. During real-time voice data processing, the system first determines whether the robot is in a voice interaction-enabled state. When the robot is awake, the system preprocesses the collected audio data, converts it into a standard voice data format with a preset sampling rate, and then sends it to the cloud. When the robot is silent, the system cancels the upload or processing of the current audio data to avoid generating invalid voice requests in a non-interactive state.

[0026] S102: Convert the preprocessed audio data into text information, perform semantic analysis on the text information, determine the task type, extract the target object, task parameters and contextual information, and generate structured semantic results; In practice, this step is used to achieve speech content understanding and intent parsing. After receiving the sent speech data, the speech content is recognized and converted into text information. Subsequently, semantic analysis is performed on the recognized text content to determine the task type corresponding to the user's current speech request, and the target object, task parameters, and context information required for task execution are extracted.

[0027] For voice requests with clear execution goals, such as "Take me to the nearest pharmacy," the system extracts "pharmacy" as the target location information. When a user requests "Turn 90 degrees to the left" or "Move forward 5 steps," the system extracts the corresponding motion control parameters as the basis for task execution. For complex voice requests with multiple execution goals, the system breaks down and parses the request content, extracting task information for each sub-task. For example, when a user requests "Take me to the nearest pharmacy and tell me about the road conditions," the system identifies two sub-tasks: navigation and environmental awareness. It extracts target location information and environmental analysis requirements, generating corresponding sub-task information. For voice requests that rely on historical interaction results, the system correlates and parses the request content with current conversation information. For example, when a user requests "Navigate to the first location" after completing "Find nearby pharmacies," the system determines the target object corresponding to "the first location" based on historical interaction results and uses that target object as the execution target of the current navigation task. After completing semantic parsing, the system generates structured semantic results. The structured semantic result includes at least the task type, target object, task parameters, and contextual information. The structured semantic result is then sent to the state management and task orchestration module for subsequent task requirement generation, state management, and capability orchestration processing.

[0028] S103: Generate task requirements based on the structured semantic results. The task requirements include task identifier, task type, task objective, task parameters, task status, and capability requirement information. Establish state management information associated with the session. The state management information includes session status, task status, capability call status, and execution feedback status. For tasks involving environmental awareness, generate a unique task identifier for each task and bind and store the collected environmental data with the current task identifier. Further, the task status includes created, pending execution, executing, completed, and terminated. When a stop command is received, the task control center updates the task status of the corresponding running task to terminated to prevent the current task from continuing to execute.

[0029] In practical implementation, this step is used to generate task requirements and manage state. After receiving the structured semantic results generated in S102, the system generates corresponding task requirements based on the task type, target object, task parameters, and contextual information. The task requirements describe the execution information needed to complete the current user request, including at least a task identifier, task type, task objective, task parameters, task state, and capability requirement information. In some tasks, the task requirements may also include execution conditions, session association information, and contextual reference information. This achieves the conversion of user semantic requests into robot-executable task requests. Specifically, the task identifier uniquely identifies the current task and is used for task tracking during subsequent task orchestration, execution, and state updates; the task objective describes the ultimate goal that the current task needs to achieve; the task state records the current stage of the task; and the capability requirement information describes the types of capabilities required to complete the current task.

[0030] For example, when a user requests "turn 90 degrees to the left," the system generates a motion control task requirement. This task requirement is a control task, the objective is to turn left, the parameter is a 90-degree turn, the task state is initialized to "pending execution," and the capability requirement is motion control. For complex voice requests with multiple execution objectives, the system generates multiple task requirements and associates them with the same session. For example, when a user requests "take me to the nearest pharmacy," the system generates query, navigation, and motion control task requirements based on structured semantic results. The objective is to reach the target pharmacy, the parameters include the target category "pharmacies" and current location information, and the task state is initialized to "pending execution." Since "the nearest pharmacy" is a dynamic objective rather than a fixed navigation point, the task requirement further generates execution conditions: it must be able to correctly obtain its current location information; otherwise, the task state is updated to "terminated," and corresponding feedback information is generated. Simultaneously, the system generates corresponding capability requirement information based on the task execution objective, including information query, positioning and navigation, and motion control.

[0031] After generating task requirements, the system establishes state management information associated with the current session and maintains the session state, task state, capability invocation state, and execution feedback state in a unified manner. Specifically, the session state records the current session identifier, historical interaction information, and task association information; the task state records the task creation, pending execution, in progress, completed, and terminated states; the capability invocation state records real-time operational data generated during task execution, including robot real-time localization results, environmental analysis results, data update time, and whether the capability invocation was successful; and the execution feedback state records the task feedback results and feedback completion status.

[0032] During task execution, the system continuously receives information from each capability module and updates the corresponding status data in real time. The system determines the relationship between the current voice request and historical tasks, and uses historical status information to complete, update, or constrain the current task requirements. For example, if a user requests "navigate to the first location" after completing "find nearby pharmacies," the system does not recreate the query task requirement. Instead, it determines the target object corresponding to the "first location" based on the candidate location information saved in the session state and writes it into the task objective or task parameters in the current task requirement. When a user requests "stop navigation" during navigation, the system updates the current navigation task status to terminated, thereby preventing the current task from continuing.

[0033] Through the aforementioned state management method, the system can establish the association between the current task and historical tasks during continuous voice interaction, and dynamically adjust task requirements and subsequent execution processes based on state changes. This enables continuous transmission of task information and state inheritance during cross-turn voice interaction, avoiding the repeated creation and execution of the same task. It also enables state sharing and collaborative processing among multiple capability modules, improving the continuity, consistency, and cross-turn interaction processing capabilities of task execution in complex voice interaction scenarios.

[0034] S104: Generate a task execution strategy based on the task requirements and the status management information; when the capability requirement information involves dynamic target location query, call the information query module to query a set of candidate locations based on the current positioning information and target category, feed the set of candidate locations back to the user for confirmation, determine the final navigation target location based on the user's confirmation result, and generate navigation task requirements; when the capability requirement information involves multiple capabilities, establish a capability execution chain, determine the calling order according to the data dependency relationship between capabilities, and call the robot capability module to execute the task; during execution, only the environmental data bound to the current task identifier is called to participate in the analysis and processing; In practical implementation, this step is used to achieve task orchestration and capability invocation. After receiving the task requirements generated in S103, the system generates a corresponding task execution strategy based on the task requirements and current status information, and invokes the corresponding capabilities to participate in task execution according to the task execution strategy. Unlike the traditional approach of processing user requests independently according to functional modules, this embodiment uses task requirements as the core of capability collaborative execution. Based on information such as the task objective and capability requirements in the task requirements, the system determines the capabilities participating in the current task execution and establishes collaborative relationships between the capabilities, enabling multiple capabilities to be invoked around the same task objective. For tasks that can be completed using only a single capability, the system directly invokes the corresponding capability to complete the task execution.

[0035] For example, when a user requests "turn 90 degrees to the left," the capability requirements in the task description only include motion control. The system directly calls the motion control module to generate the corresponding control command and execute the task. For tasks requiring the coordinated participation of multiple capabilities, the system establishes a capability execution chain based on the capability requirements in the task description and determines the calling order according to the data dependencies between capabilities. During the establishment of the capability execution chain, the system first parses the input and output information corresponding to each capability; then, it establishes capability relationships based on the data dependencies between capabilities; when the output of one capability serves as the input condition for another capability, the system establishes the corresponding data transmission relationship and determines the capability calling order according to the data generation order, thus forming a capability execution chain corresponding to the current task. For example, when a user requests "take me to the nearest pharmacy," the capability requirements in the task description include information query capability, positioning and navigation capability, and motion control capability. The system first calls the information interpolation module to obtain candidate location information; then it provides the candidate location information as input to the positioning and navigation capability to determine the navigation target and generate a navigation path; then it provides the navigation path as input to the motion control capability to control the robot to perform navigation tasks according to the planned path, thus forming a capability execution chain in which information query, positioning and navigation and motion control are executed in sequence.

[0036] For complex tasks involving multiple capability coordination requirements, the system can dynamically expand the capability execution chain according to the task requirements. For example, when a user requests "Take me to the nearest pharmacy and tell me about the route," the system simultaneously invokes the environmental awareness module during the navigation task execution and associates the result with the current task state, thereby achieving coordinated processing of navigation task execution and environmental information feedback. After generating the task execution strategy, the system invokes the corresponding capability modules to execute the task according to the capability execution chain, entering the corresponding capability processing processes from S104a to S104d.

[0037] S104a Environment Awareness Module. Calling the environment awareness module includes: Image data acquired by visible light cameras and depth cameras is obtained and associated with the unique task identifier. In multiple consecutively initiated environment viewing requests, the corresponding environment data is distinguished based on different task identifiers, and only the image data associated with the current task identifier is input into the visual big model for analysis.

[0038] When a task requires environmental observation, road condition assessment, target recognition, or scene analysis, the system invokes the environmental perception module to participate in task execution. Environmental perception data includes camera image data, LiDAR point cloud data, and data from other environmental sensors. The system determines the environmental analysis target based on the task objectives and parameters in the task requirements and associates the current environmental data with the current task. For example, when a user requests "Help me see what's ahead" or "Can I pass through here?", the environmental perception module acquires the current camera image and performs scene recognition, target detection, or road condition analysis according to the task requirements, generating corresponding analysis results. To ensure the analysis results remain consistent with the current scene, the system establishes a unique task identifier before the environmental perception module executes and associates the currently acquired environmental data with this task identifier. During task execution, unless the user explicitly specifies historical data, only environmental data associated with the current task identifier is allowed to participate in the analysis. When a user initiates multiple environmental observation requests consecutively, the system distinguishes the environmental data corresponding to different tasks based on the task identifier and only uses the environmental data associated with the current task to generate analysis results, thus avoiding the influence of historical environmental information on the current analysis results. After the environmental analysis is completed, the environmental perception module generates environmental description results, target recognition results, or road traffic status results, and returns the analysis results after associating them with the current task identifier.

[0039] S104b Positioning and Navigation Module. Calling the positioning and navigation module specifically includes: Obtain positioning information including latitude and longitude, positioning timestamp, positioning status, and positioning accuracy; stop navigation processing when the positioning status is abnormal or the positioning accuracy is lower than a preset threshold; for scenarios involving generalized navigation requests, supplement the current task requirements with the robot's current valid positioning information as query conditions.

[0040] When the task requirements involve reaching a target location, autonomous navigation, or path planning, the system invokes the positioning and navigation module to participate in task execution. The positioning and navigation module obtains the robot's current positioning information and determines whether the navigation execution conditions are met based on this information. The positioning information includes latitude and longitude, positioning timestamp, positioning status, and positioning accuracy. When the positioning status is abnormal or the positioning accuracy is lower than a preset threshold, the system stops the current navigation process and generates corresponding positioning error feedback information. For scenarios where the user directly specifies a target location, the system invokes the information query module to directly search for the latitude and longitude information of the target location and writes the query results into the target information in the current task requirements. The positioning and navigation module obtains the target location according to the task requirements and executes subsequent navigation tasks. For scenarios where the user submits generalized navigation requests such as "nearest subway station," "nearby bus stop," or "nearby supermarket," the system first obtains the robot's current valid positioning information and uses it as a query condition to supplement the current task requirements. Subsequently, the information query module performs a surrounding search based on the target category information in the task requirements and the current location information, obtains candidate location information that meets the conditions, generates a candidate location set, and sends it back to the user for confirmation. Finally, the system determines the final navigation target location based on the user's confirmation and generates new task requirements. After determining the navigation target location, the positioning and navigation module executes the navigation task according to the specified target location scenario.

[0041] S104c Motion Control Module. Calling the motion control module specifically includes: The robot receives motion control requests directly from users or navigation paths generated by the positioning and navigation module; it dynamically generates motion control commands based on the control parameters in the motion control request or the navigation path, and sends them to the motor driver through the robot's underlying control interface; during movement, it adjusts the robot's motion state in real time based on obstacle information obtained by lidar and ultrasonic ranging sensors.

[0042] When a task requires robot motion execution, the system invokes the motion control module to participate in the task. The motion control module generates corresponding motion control commands based on the task requirements and sends them to the robot's motion control system via the robot's underlying control interface. For motion control requests directly issued by the user, the motion control module generates corresponding motion control commands based on the control parameters in the task execution strategy. Examples include "turn 90 degrees left," "move forward 5 steps," and "stop moving." For motion requirements generated by navigation tasks, the motion control module dynamically generates motion control commands based on the navigation path and navigation status information generated by the positioning and navigation module. During navigation execution, the motion control module also adjusts the robot's direction and state in real time based on obstacle information. Through this method, the motion control module uniformly converts user motion control requests or motion requirements generated by navigation tasks into executable motion control commands for the robot, achieving a seamless connection between task requirements and the robot's underlying motion control.

[0043] S104d Information Query Module. Calling the information query module specifically includes: Based on the target category information in the task execution strategy and the current location information, perform location retrieval processing; generate a set of candidate locations containing destination names and corresponding latitude and longitude information; and write the query results into the current task requirements and historical session information for subsequent task calls.

[0044] When a task requires the acquisition of external information, the system invokes the information query module to participate in task execution. Typically, the query content includes weather information, location service information, and other internet information. For weather queries, location queries, and other information service tasks, the information query module accesses the corresponding information service interface based on the current task requirements. For navigation-related tasks, the information query module performs location retrieval processing based on the target category information in the task execution strategy and the current location information, and obtains candidate location information that meets the criteria based on the target category and the current location, generating a candidate location set. The candidate location information includes at least the destination name and corresponding latitude and longitude information for use in subsequent navigation processing. After the query is completed, the information query module writes the results into the current task requirements or task status information. Through the above methods, the information query module realizes data interaction between task requirements and external information services, and provides data support for navigation tasks, information service tasks, and other tasks that rely on external data.

[0045] During task execution, the system can also respond to new voice requests from users by dynamically inserting new capability call nodes into the established capability execution chain. The system can dynamically expand the capability execution chain as needed. For example, if there was originally only a navigation task, but during execution it is discovered that the user has added a requirement for environmental description, the system can dynamically add an environmental awareness module.

[0046] S105: During task execution, continuously receive the execution results returned by the robot capability module, and update the task status in real time according to the execution results; when a voice request that depends on the historical interaction results is received, complete, update or constrain the current task requirements according to the historical task status saved in the session status; when the environment analysis is abnormal, the path fails, the target is unreachable, the positioning is abnormal, the navigation times out, or the query fails, the query times out, or the external service is abnormal, determine that the current capability call has failed, immediately stop the subsequent execution process of the current task, update the corresponding task status to terminated, and record the corresponding abnormal reason; In practice, this step is used to implement task execution and status updates. After completing task orchestration and capability invocation, the system organizes each capability module to execute the corresponding task according to the task execution strategy, and continuously receives the execution results and status information returned by each capability module during task execution. For environmental awareness tasks, the system receives the analysis results returned by the environmental awareness module and associates and stores them with the current task identifier. When the environmental analysis is completed, the system updates the task status to "completed" and writes the analysis results into the execution feedback of the current task. When the user subsequently initiates another environmental viewing request, the system re-obtains the current environmental data based on the new task identifier, instead of directly reusing historical analysis results, to avoid the impact of historical analysis results on the current task.

[0047] When environmental data acquisition fails or environmental analysis is abnormal, the system determines that the current capability call has failed, immediately stops the subsequent execution of the current task, updates the task status to terminated, records the corresponding reason for the exception, and generates exception feedback information. For navigation tasks, the system continuously receives navigation status information returned by the positioning and navigation module and updates the current task status according to changes in the navigation status. When the navigation target is determined, the system updates the task status to in progress and records the navigation path information; during navigation execution, the system continuously updates the navigation progress information and the current position information. When the robot reaches the target location, the system updates the task status to completed and records the navigation completion result. When abnormal situations such as path failure, target unreachability, positioning abnormality, or navigation timeout occur during navigation, the system stops the current navigation task, updates the task status to terminated, and records the corresponding reason for the exception.

[0048] For motion control tasks, the task status is updated to "completed" upon completion of the motion control command; when a control anomaly, motion obstruction, or task termination occurs during motion, the task status is updated to "terminated," and the corresponding anomaly information is recorded. Employing an anomaly-driven state termination mechanism, the system not only updates the status upon task completion but also forcibly updates the task status to "terminated" and stops subsequent processes when specific anomalies are detected (such as positioning accuracy below a threshold, environmental analysis failure, or external service timeout). This prevents the robot from continuing to perform dangerous actions in abnormal states.

[0049] For information query tasks, when the query is successfully completed, the system updates the task status to "completed" and writes the query results into the current task execution results and historical session information. The system can then directly obtain the corresponding information to complete the task requirements. When the query fails, times out, or an external service fails, the system stops the current query task, updates the task status to "terminated," and records the exception information.

[0050] Through the aforementioned state update mechanism, the system achieves real-time correlation between the execution results of each capability and the task status, and uses state changes to drive subsequent task processing and execution feedback generation, avoiding problems such as duplicate execution, erroneous execution, and inconsistencies between feedback results and actual execution status.

[0051] S106: The execution results are filtered and fused to generate feedback text. The text-to-speech module is used to synthesize speech and broadcast it. During the broadcast, the microphone is controlled to stop collecting or pause audio data uploading through a message communication mechanism. After the broadcast is completed, voice input processing is resumed and the current feedback status is updated.

[0052] Furthermore, the filtering of execution results includes: The system identifies the timestamp of the execution result and its associated task identifier; removes expired feedback information whose timestamp is earlier than the current task status update time; and filters execution results that do not match the current task status and duplicate execution results generated by module calls. The system proactively filters "expired historical execution results." For example, after navigation begins, old messages like "Start Navigation" will no longer be played. This solves the problem of overloaded and outdated feedback information in multi-turn interactions.

[0053] The generated feedback text includes: For single-capability tasks, the feedback content is generated directly from the execution result returned by the corresponding capability; for multi-capability collaborative tasks, a unified feedback content is generated based on the correlation between the results returned by each capability; during the execution of a navigation task, the navigation execution status returned by the positioning and navigation module is fused with the environmental description information returned by the environmental perception module to generate the feedback content.

[0054] In practice, this step is used for execution result feedback and voice broadcasting. The system receives the execution results generated during the execution of task S105 and generates corresponding feedback content based on the current task status. During the feedback content generation process, the system first performs association matching on the execution results based on the task identifier to determine the task to which the execution result belongs and the corresponding session. For multiple feedback results generated during the execution of the same task, the system filters them based on the current task status. Historical execution results that have expired, execution results that do not match the current task status, and duplicate execution results generated by module calls are no longer included in the subsequent feedback content generation. For example, when a navigation task has been completed, historical progress feedback during the navigation process (such as "Start Navigation") is no longer included in the final feedback result generation. After completing the filtering process, the system merges the remaining execution results according to the task type. For single-capability tasks, the system directly uses the execution result returned by the corresponding capability to generate feedback content; for multi-capability collaborative tasks, the system generates unified feedback content based on the correlation between the results returned by each capability.

[0055] For example, when a user requests, "Take me to the nearest pharmacy and tell me about the route," the system simultaneously obtains the execution results from both the positioning and navigation module and the environmental perception module. It then merges the navigation execution results with the environmental description information to generate feedback. After generating the feedback, the system sends the feedback text to the text-to-speech (TTS) module for speech synthesis and broadcasts it to the user through the robot's speaker. During the speech broadcast, the system pauses speech recognition processing or speech upload processing depending on the current broadcast status to prevent the robot's own broadcast content from being captured by the microphone again and triggering a new speech recognition process. Once the broadcast is complete, the system resumes speech input processing and updates the current feedback status.

[0056] Through the above-mentioned execution result processing and voice broadcasting mechanism, the system realizes the conversion of task execution results into user-understandable feedback content, improving the accuracy and reliability of feedback results in complex voice interaction scenarios.

[0057] To facilitate the implementation of the above method, such as Figure 2 As shown, the system includes a speech recognition and conversation management module, a speech content understanding and intent parsing module, a task requirement generation and state management module, a task orchestration and capability invocation module, a robot capability module, a task execution and state update module, and an execution result feedback and voice broadcast module. Among these, the task requirement generation and state management module serves as the core control module, internally containing a task control center for maintaining task status, coordinating information exchange between functional modules, and saving task execution results. The robot capability module includes a localization and navigation module, an environmental perception module, a motion control module, and an information query module.

[0058] The positioning and navigation module includes an RTK positioning module, a lidar, and an ultrasonic ranging sensor; the environmental perception module includes a visible light camera and a depth camera; the motion control module includes a motor driver and an IMU attitude sensor; and the information query module includes a WiFi communication module and a 4G communication module. Each capability module is uniformly scheduled by the system to complete the corresponding task execution.

[0059] During system operation, the speech recognition and session management module first collects user speech data, resamples the speech data, and establishes and maintains a session relationship with the cloud-based large model. Subsequently, the recognized speech commands are input into the speech content understanding and intent parsing module. The cloud-based language model and cloud-based vision model perform semantic understanding of the user's speech request, determine the task type corresponding to the current speech request, and parse the target information, parameter information, and contextual information required for task execution to generate structured semantic results.

[0060] After receiving the structured semantic results, the task requirement generation and status management module generates corresponding task requirements based on the parsing results and establishes status information corresponding to the current task. The task control center maintains the task status based on the current task requirements and determines whether the current task execution conditions are met by combining the status information. The task orchestration and capability invocation module generates corresponding task execution strategies based on the task requirements and status information maintained by the task control center and establishes a capability execution chain. When a task involves only a single capability, the corresponding capability is directly invoked to complete the task execution; when a task involves the collaborative processing of multiple capabilities, the task orchestration and capability invocation module organizes multiple capabilities to be executed sequentially according to the data dependencies between capabilities, and achieves unified management of task status and execution information among the capabilities through the task control center.

[0061] For example, when a user request involves tasks such as environmental viewing, target recognition, or scene understanding, the task orchestration and capability invocation module calls the environmental perception module to collect information about the robot's surrounding environment and generate an environmental description result relevant to the current task. When a user request involves navigation services, the navigation and positioning module is invoked to comprehensively utilize robot positioning information, environmental perception information, and map information to analyze the robot's current position and target position, generating corresponding navigation target information and path planning results. When a user request involves robot motion control, the motion control module is invoked to generate corresponding motion control commands and send them to the robot's underlying control system through a message communication mechanism, enabling the robot to move forward, backward, turn, stop, and perform other motion control functions. When a task involves weather inquiries, point-of-interest (POI) inquiries, or internet information retrieval, the information query module is invoked to access the corresponding external information service interface to obtain the required data. During task execution, the task execution and status update module continuously receives execution results returned by each capability module and updates the task status and status information in real time based on the task execution status. When anomalies occur during capability execution, such as positioning anomalies, environmental analysis anomalies, motion control anomalies, or external service anomalies, the task status is updated to "terminated," the corresponding anomaly cause is recorded, and the status change is fed back to the task control center. The execution result feedback and voice broadcast module receives the execution results generated during task execution and generates the final feedback text based on the task status maintained by the task control center. Subsequently, the feedback text is sent to the text-to-speech (TTS) module for speech synthesis, and then broadcast through the robot's speaker, thus forming a complete multimodal voice interaction closed loop of "voice input—semantic understanding—task requirement generation—task orchestration—robot capability execution—status update—execution feedback—voice broadcast".

[0062] In summary, the multimodal voice interaction method provided by this invention constructs a task orchestration mechanism centered on task requirements, enabling each capability module to participate in task execution around the same task objective. This avoids problems such as task execution fragmentation and execution conflicts caused by independent execution of each capability module, while reducing problems such as repeated calls, repeated interactions, and repeated feedback during the execution of complex multimodal voice tasks, thereby improving the execution continuity and reliability of complex multimodal voice tasks.

[0063] By establishing a state management approach that spans the entire process of task creation, execution, termination, and feedback for complex multimodal speech tasks, the task state can be continuously updated as the task is executed. This avoids problems such as duplicate execution, erroneous execution, and inconsistencies between feedback content and actual execution state caused by lag in task state, thereby improving the execution consistency and interaction reliability of complex multimodal speech interaction processes.

[0064] By constructing a mechanism for generating task requirements, orchestrating tasks, and coordinating the execution of capabilities, the intentions expressed by users during natural language interaction are gradually converted into task semantics that the robot can directly execute, and further converted into corresponding robot actions. This achieves an effective connection between natural language interaction and actual robot actions, enabling users to directly drive the robot to complete complex tasks using natural language, thereby improving the intelligence level of complex multimodal voice interaction.

[0065] By uniformly processing, filtering, and integrating the feedback results of each capability module during robot execution, conflicts and duplicate feedback between execution results from different sources are eliminated. At the same time, feedback information that has become invalid or is inconsistent with the current task status is removed. This avoids problems such as repeated broadcasts, erroneous broadcasts, and delayed feedback in complex multimodal voice interaction, thereby improving the accuracy and consistency of robot execution result feedback.

[0066] See Figure 3 The present invention can also provide a multimodal voice interaction device, such as... Figure 3 As shown, the device may include: The speech recognition and conversation management unit 601 is used to collect audio data in pulse code modulation format through a microphone and upload the preprocessed audio data to the cloud when the robot is in a wake-up state. The speech content understanding and intent parsing unit 602 is used to convert the uploaded audio data into text information, perform semantic analysis on the text information, determine the task type, extract the target object, task parameters and contextual information, and generate structured semantic results. The task requirement generation and status management unit 603 is used to generate task requirements based on the structured semantic results. The task requirements include task identifier, task type, task objective, task parameters, task status, and capability requirement information. It also establishes status management information associated with the session, including session status, task status, capability call status, and execution feedback status. For tasks involving environmental awareness, it generates a unique task identifier for each task and binds and stores the collected environmental data with the current task identifier. The task orchestration and capability invocation unit 604 is used to generate a task execution strategy based on the task requirements and the status management information. When the capability requirement information involves dynamic target location query, the information query module is invoked to query a set of candidate locations based on the current positioning information and the target category, and the set of candidate locations is fed back to the user for confirmation. Based on the user's confirmation result, the final navigation target location is determined and a navigation task requirement is generated. When the capability requirement information involves multiple capabilities, a capability execution chain is established, the invocation order is determined according to the data dependency relationship between capabilities, and the robot capability module is invoked to execute the task. During the execution process, only the environmental data bound to the current task identifier is invoked for analysis and processing. The task execution and status update unit 605 is used to continuously receive the execution results returned by the robot capability module during task execution, and update the task status in real time according to the execution results; when a voice request that depends on the historical interaction results is received, the current task requirements are completed, updated or constrained according to the historical task status saved in the session status; when the environment analysis is abnormal, the path fails, the target is unreachable, the positioning is abnormal, the navigation times out, or the query fails, the query times out, or the external service is abnormal, the current capability call is determined to have failed, the subsequent execution process of the current task is stopped immediately, the corresponding task status is updated to terminated, and the corresponding abnormal reason is recorded. The execution result feedback and voice broadcasting unit 606 is used to filter and fuse the execution results, generate feedback text, synthesize speech using a text-to-speech module, and broadcast it. During the broadcasting process, the microphone is controlled to stop collecting or pause audio data uploading through a message communication mechanism. After the broadcasting is completed, voice input processing is resumed, and the current feedback status is updated.

[0067] This invention can also provide a multimodal voice interaction device, the device including a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the steps of the multimodal voice interaction method described above according to the instructions in the program code.

[0068] like Figure 4 As shown in the figure, a multimodal voice interaction device provided in this embodiment of the invention may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, the memory 11, and the communication interface 12 all communicate with each other through the communication bus 13.

[0069] In this embodiment of the invention, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0070] The processor 10 can call programs stored in the memory 11. Specifically, the processor 10 can execute operations in the embodiments of the multimodal voice interaction method.

[0071] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment of the invention, the memory 11 stores at least a program for implementing the following functions: Audio data in pulse code modulation format is collected through a microphone. When the robot is in a wake-up state, the audio data is preprocessed and then uploaded to the cloud. The preprocessed audio data is converted into text information, and semantic analysis is performed on the text information to determine the task type, extract the target object, task parameters and contextual information, and generate structured semantic results. Based on the structured semantic results, task requirements are generated, including task identifier, task type, task objective, task parameters, task status, and capability requirement information; state management information associated with the session is established, including session status, task status, capability invocation status, and execution feedback status; for tasks involving environmental awareness, a unique task identifier is generated for each task, and the collected environmental data is bound and stored with the current task identifier. A task execution strategy is generated based on the task requirements and the status management information. When the capability requirement information involves dynamic target location query, the information query module is invoked to query a set of candidate locations based on the current location information and the target category. The set of candidate locations is then fed back to the user for confirmation. Based on the user's confirmation, the final navigation target location is determined, and a navigation task requirement is generated. When the capability requirement information involves multiple capabilities, a capability execution chain is established. The calling order is determined according to the data dependencies between capabilities, and the robot capability module is invoked to execute the task. During execution, only environmental data bound to the current task identifier is called for analysis and processing. During task execution, the robot continuously receives execution results returned by the robot capability module and updates the task status in real time based on the execution results. When a voice request that depends on historical interaction results is received, the robot completes, updates, or constrains the current task requirements based on the historical task status saved in the session status. When there are abnormalities in environmental analysis, path failure, target unreachability, positioning abnormalities, navigation timeouts, query failures, query timeouts, or external service abnormalities, the robot determines that the current capability call has failed, immediately stops the subsequent execution process of the current task, updates the corresponding task status to terminated, and records the corresponding abnormal reason. The execution results are filtered and fused to generate feedback text. The text-to-speech module is used to synthesize speech and broadcast it. During the broadcast, the microphone is controlled to stop collecting or pause audio data uploading through a message communication mechanism. After the broadcast is completed, voice input processing is resumed and the current feedback status is updated.

[0072] In one possible implementation, the memory 11 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (such as file creation or data read / write). The data storage area may store data created during use, such as initialization data.

[0073] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0074] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.

[0075] Of course, it should be noted that, Figure 4 The structure shown does not constitute a limitation on the multimodal voice interaction device in the embodiments of the present invention. In practical applications, the multimodal voice interaction device may include more than Figure 4 More or fewer components as shown, or combinations of certain components.

[0076] This invention also provides a computer-readable storage medium for storing program code for executing the steps of the multimodal voice interaction method described above.

[0077] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0078] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0079] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0080] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A multimodal voice interaction method, characterized in that, The method includes: Audio data in pulse code modulation format is collected through a microphone. When the robot is in a wake-up state, the audio data is preprocessed and then uploaded to the cloud. The preprocessed audio data is converted into text information, and semantic analysis is performed on the text information to determine the task type, extract the target object, task parameters and contextual information, and generate structured semantic results. Based on the structured semantic results, task requirements are generated, including task identifier, task type, task objective, task parameters, task status, and capability requirement information; state management information associated with the session is established, including session status, task status, capability invocation status, and execution feedback status; for tasks involving environmental awareness, a unique task identifier is generated for each task, and the collected environmental data is bound and stored with the current task identifier. A task execution strategy is generated based on the task requirements and the status management information. When the capability requirement information involves dynamic target location query, the information query module is invoked to query a set of candidate locations based on the current location information and the target category. The set of candidate locations is then fed back to the user for confirmation. Based on the user's confirmation, the final navigation target location is determined, and a navigation task requirement is generated. When the capability requirement information involves multiple capabilities, a capability execution chain is established. The calling order is determined according to the data dependencies between capabilities, and the robot capability module is invoked to execute the task. During execution, only environmental data bound to the current task identifier is called for analysis and processing. During task execution, the robot continuously receives execution results returned by the robot capability module and updates the task status in real time based on the execution results. When a voice request that depends on historical interaction results is received, the robot completes, updates, or constrains the current task requirements based on the historical task status saved in the session status. When there are abnormalities in environmental analysis, path failure, target unreachability, positioning abnormalities, navigation timeouts, query failures, query timeouts, or external service abnormalities, the robot determines that the current capability call has failed, immediately stops the subsequent execution process of the current task, updates the corresponding task status to terminated, and records the corresponding abnormal reason. The execution results are filtered and fused to generate feedback text. The text-to-speech module is used to synthesize speech and broadcast it. During the broadcast, the microphone is controlled to stop collecting or pause audio data uploading through a message communication mechanism. After the broadcast is completed, voice input processing is resumed and the current feedback status is updated.

2. The multimodal voice interaction method according to claim 1, characterized in that, The preprocessing includes resampling and converting the audio data into a standard speech data format with a preset sampling rate; the cloud includes a cloud-based language model for speech recognition and a cloud-based visual model for image analysis.

3. The multimodal voice interaction method according to claim 1, characterized in that, The task status includes created, pending execution, executing, completed, and terminated; when a stop command is received, the task control center updates the task status of the corresponding running task to terminated to prevent the current task from continuing to execute.

4. The multimodal voice interaction method according to claim 1, characterized in that, Calling the environment awareness module includes: Image data acquired by visible light cameras and depth cameras is obtained and associated with the unique task identifier. In multiple consecutively initiated environment viewing requests, the corresponding environment data is distinguished based on different task identifiers, and only the image data associated with the current task identifier is input into the visual big model for analysis.

5. The multimodal voice interaction method according to claim 1, characterized in that, Calling the location and navigation module specifically includes: Obtain positioning information including latitude and longitude, positioning timestamp, positioning status, and positioning accuracy; stop navigation processing when the positioning status is abnormal or the positioning accuracy is lower than a preset threshold; for scenarios involving generalized navigation requests, supplement the current task requirements with the robot's current valid positioning information as query conditions.

6. The multimodal voice interaction method according to claim 1, characterized in that, Calling the motion control module specifically includes: The robot receives motion control requests directly from users or navigation paths generated by the positioning and navigation module; it dynamically generates motion control commands based on the control parameters in the motion control request or the navigation path, and sends them to the motor driver through the robot's underlying control interface; during movement, it adjusts the robot's motion state in real time based on obstacle information obtained by lidar and ultrasonic ranging sensors.

7. The multimodal voice interaction method according to claim 1, characterized in that, The information query module includes the following: Based on the target category information in the task execution strategy and the current location information, perform location retrieval processing; generate a set of candidate locations containing destination names and corresponding latitude and longitude information; and write the query results into the current task requirements and historical session information for subsequent task calls.

8. The multimodal voice interaction method according to claim 1, characterized in that, Filtering the execution results includes: Identify the timestamp of the execution result and the associated task identifier; remove expired feedback information whose timestamp is earlier than the current task status update time; filter execution results that do not match the current task status and duplicate execution results generated by module calls.

9. The multimodal voice interaction method according to claim 1, characterized in that, The generated feedback text includes: For single-capability tasks, the feedback content is generated directly from the execution result returned by the corresponding capability; for multi-capability collaborative tasks, a unified feedback content is generated based on the correlation between the results returned by each capability; during the execution of a navigation task, the navigation execution status returned by the positioning and navigation module is fused with the environmental description information returned by the environmental perception module to generate the feedback content.

10. A multimodal voice interaction device, characterized in that, The apparatus for performing the multimodal voice interaction method according to any one of claims 1-9, the apparatus comprising: The speech recognition and conversation management unit is used to collect audio data in pulse code modulation format through a microphone and upload the preprocessed audio data to the cloud when the robot is in a wake-up state. The speech content understanding and intent parsing unit is used to convert the uploaded audio data into text information, perform semantic analysis on the text information, determine the task type, extract the target object, task parameters and contextual information, and generate structured semantic results. The task requirement generation and status management unit is used to generate task requirements based on the structured semantic results. The task requirements include task identifier, task type, task objective, task parameters, task status, and capability requirement information. It also establishes status management information associated with the session, including session status, task status, capability call status, and execution feedback status. For tasks involving environmental awareness, it generates a unique task identifier for each task and binds and stores the collected environmental data with the current task identifier. The task orchestration and capability invocation unit is used to generate a task execution strategy based on the task requirements and the status management information. When the capability requirement information involves dynamic target location query, the information query module is invoked to query a set of candidate locations based on the current positioning information and the target category. The set of candidate locations is then fed back to the user for confirmation. Based on the user's confirmation, the final navigation target location is determined and a navigation task requirement is generated. When the capability requirement information involves multiple capabilities, a capability execution chain is established. The invocation order is determined according to the data dependencies between capabilities, and the robot capability module is invoked to execute the task. During execution, only environmental data bound to the current task identifier is invoked for analysis and processing. The task execution and status update unit is used to continuously receive the execution results returned by the robot capability module during task execution, and update the task status in real time according to the execution results; when a voice request that depends on the historical interaction results is received, the current task requirements are completed, updated or constrained according to the historical task status saved in the session status; when the environment analysis is abnormal, the path fails, the target is unreachable, the positioning is abnormal, the navigation times out, or the query fails, the query times out, or the external service is abnormal, the current capability call is determined to have failed, the subsequent execution process of the current task is stopped immediately, the corresponding task status is updated to terminated, and the corresponding abnormal reason is recorded. The execution result feedback and voice broadcasting unit is used to filter and fuse the execution results, generate feedback text, synthesize speech using a text-to-speech module, and broadcast it. During the broadcasting process, the microphone is controlled to stop collecting or pause audio data uploading through a message communication mechanism. After the broadcasting is completed, voice input processing is resumed, and the current feedback status is updated.

Images