Robot dynamic voice interaction method and system based on multi-modal state management

Abstract

The invention discloses a robot dynamic voice interaction method based on multi-modal state management, and belongs to the technical field of intelligent robots, and the method comprises the steps: S1, collecting multi-modal instruction data inputted by a user; s2, carrying out data processing on the multi-mode instruction data, and then generating an input text packet with an environment context mark; s3, performing intention analysis on the input text packet, and binding the input text packet with a historical context to generate an intention vector; s4, updating the environment matrix in real time through the intention vector; s5, dynamically calculating the task weight W through the updated environment matrix, and dynamically adjusting the priority of each task; and S6, calling a corresponding function module according to the priority of each task through a plug-in interface, and executing the corresponding task, so as to achieve the purposes of realizing cross-scene context continuity, multi-task priority dynamic scheduling, supporting hot plug function extension and having active abnormal interaction capability.

Description

Technical Field

[0001] This invention belongs to the technical field of intelligent robots, and more specifically, relates to a dynamic voice interaction method and system for robots based on multimodal state management. Background Technology

[0002] Voice-interactive robots are intelligent devices or systems that enable human-computer interaction through technologies such as speech recognition and natural language processing. They are mainly used to provide services such as information retrieval and command execution.

[0003] Existing voice-interactive robots mainly suffer from the following shortcomings:

[0004] (1) Context fragmentation: Traditional voice robots (such as smart speakers) cannot continuously track user intent across scenarios. Users need to repeatedly wake up the device in different tasks or environments, resulting in low interaction efficiency.

[0005] (2) Multi-task conflict: When multiple functional modules (such as navigation and object recognition) run in parallel, the lack of a unified priority arbitration mechanism can easily lead to chaotic task execution or delay of critical tasks.

[0006] (3) Static architecture: Most existing voice interaction robots adopt a fixed modular architecture. When expanding new functions, it is necessary to make significant modifications to the code or hardware interface, which lacks flexible scalability.

[0007] (4) Lack of abnormal handling: When encountering sudden environmental events (such as obstacles, people falling, fires, etc.), the existing voice interaction robots cannot actively trigger voice interaction or adjust the current task process.

[0008] Based on the above problems, there is an urgent need to develop a robot dynamic voice interaction system that can achieve cross-scenario context continuity, dynamic scheduling of multi-task priorities, support for hot-swappable function expansion, and has the ability to actively interact with abnormal situations. Summary of the Invention

[0009] In view of this, in order to solve the above-mentioned problems of the prior art, the purpose of the present invention is to provide a robot dynamic voice interaction method and system based on multimodal state management to achieve cross-scene context continuity, dynamic scheduling of multi-task priorities, support for hot-swappable function expansion, and proactive abnormal interaction capability.

[0010] The technical solution adopted in this invention is: a robot dynamic voice interaction method based on multimodal state management, the method comprising:

[0011] S1: Collect multimodal command data input by the user;

[0012] S2: After processing the multimodal instruction data, generate an input text package with environment context markers;

[0013] S3: After parsing the intent of the input text packet, bind it with the historical context to generate an intent vector;

[0014] S4: Update the environment matrix in real time using intent vectors;

[0015] S5: Calculate the task weight W dynamically using the updated environment matrix, and dynamically adjust the priority of each task;

[0016] S6: Through the plug-in interface, the corresponding functional modules are called according to the priority of each task, and the corresponding tasks are executed.

[0017] Furthermore, the multimodal instruction data includes: user voice commands, text information, and environmental sensor data, and the environmental sensor data includes: image acquisition data, lidar data, temperature and humidity data, and infrared detection data.

[0018] Furthermore, the data processing method for the multimodal command data is as follows:

[0019] S201: The voice command is recognized and converted by the ASR module to generate structured text;

[0020] S202: The environmental sensor data is timestamped and aligned with the structured text through the data fusion module;

[0021] S203: Generate an input text package with environment context tags.

[0022] Furthermore, the method for generating the intent vector is as follows:

[0023] S301: The natural language processing unit performs semantic understanding on the input text packet to generate a preliminary intent vector;

[0024] S302: Bind the initial intent vector to the historical context of the state manager;

[0025] S303: Generate an intent vector containing spatial parameters.

[0026] Furthermore, the intent vector is read through the state manager, and the environment matrix is ​​updated in real time.

[0027] Furthermore, the calculation formula for the task weight W is: W = α * user urgency + β * security level + γ * energy consumption coefficient + δ * module dependency; where α, β, γ and δ are constants.

[0028] Furthermore, in S6, the execution progress and results of the task are sent back to the state manager, and natural language is generated and fed back to the user through the speech synthesis module.

[0029] Furthermore, the method also includes: real-time monitoring of the environmental matrix; if an anomaly is detected, the current task is interrupted and active voice interaction is triggered; if no anomaly is detected, the process continues in a loop.

[0030] This invention also discloses a robot dynamic voice interaction system based on multimodal state management, the system comprising:

[0031] A multimodal input module, wherein the multimodal input module is used to receive multimodal command data input by the user;

[0032] An intent parsing module is communicatively connected to a multimodal input module, the intent parsing module being used to generate an intent vector containing spatial parameters;

[0033] A state manager that communicates with the intent parsing module, the state manager being used to dynamically update the environment matrix;

[0034] An intent dispatcher that communicates with a state manager calculates task priority weights based on an environment matrix and arbitrates resource allocation for tasks according to the priority weights;

[0035] A pluggable function library that communicates with the intent dispatcher includes multiple independently callable function plugins, each of which is used to perform a corresponding task.

[0036] Furthermore, the intent distributor and the state manager form a dual-central collaborative architecture through real-time data exchange.

[0037] The beneficial effects of this invention are as follows:

[0038] 1. The robot dynamic voice interaction method based on multimodal state management provided by this invention realizes cross-scene and cross-dialogue intent association by parsing the intent of the input text packet and binding it with the historical context in the state manager. The user does not need to wake up repeatedly and has context continuity.

[0039] 2. The robot dynamic voice interaction method based on multimodal state management provided by this invention dynamically calculates the task weight W through the environment matrix, dynamically adjusts the priority of each task, and ensures the priority execution of core functions according to the weight arbitration mechanism, which has high robustness.

[0040] 3. The robot dynamic voice interaction method based on multimodal state management provided by this invention monitors the environmental matrix in real time. If an abnormality occurs in the environmental sensing data, a voice alarm will be automatically triggered without user intervention, thus providing good proactive security. Attached Figure Description

[0041] Figure 1 This is an overall flowchart of the robot dynamic voice interaction method based on multimodal state management provided by the present invention;

[0042] Figure 2 This is a system architecture diagram of the robot dynamic voice interaction system based on multimodal state management provided by the present invention. Detailed Implementation

[0043] The embodiments of this application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar modules or modules having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. Rather, the embodiments of this application include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.

[0044] Example 1

[0045] like Figure 1 As shown, this embodiment provides a robot dynamic voice interaction method based on multimodal state management, aiming to achieve context continuity across scenarios, dynamic scheduling of multi-task priorities, support for hot-swappable function expansion, and proactive abnormal interaction capabilities. This robot dynamic voice interaction method includes:

[0046] S1: The system collects multimodal command data input by the user. The multimodal command data includes: user voice commands, text information, and environmental sensor data. The environmental sensor data includes: image data collected by an image acquisition device (such as gestures and scene images captured by a camera), radar ranging data collected by a lidar, temperature and humidity data collected by a temperature and humidity sensor, and infrared detection data collected by an infrared detector. The audio signal of the user's voice commands is preprocessed by an adaptive noise reduction processing model to eliminate environmental noise interference. The adaptive noise reduction processing model adopts the ANC algorithm of a deep neural network.

[0047] S2: After processing the multimodal command data, an input text packet with environment context markers is generated; the data processing method for the multimodal command data is as follows:

[0048] S201: The voice command is recognized and converted by the ASR module to generate structured text. The ASR module is an algorithm module based on automatic speech recognition technology.

[0049] S202: The environmental sensor data is timestamped and aligned with the structured text using the data fusion module;

[0050] S203: Generate an input text packet with an environment context marker, for example: input text packet = {text command, sensor snapshot}.

[0051] S3: After parsing the input text packet for intent, it is bound to the historical context to generate an intent vector; the method for generating the intent vector is as follows:

[0052] S301: The input text packet is semantically understood through the Natural Language Processing (NLP) unit to generate a preliminary intent vector;

[0053] S302: Bind the initial intent vector to the historical context of the state manager to achieve cross-scene and cross-dialogue intent association; for example: if the user first says "go to the kitchen" as the historical context, and then says "get an apple", the system will automatically infer the intent vector of "go to the kitchen to get an apple";

[0054] S303: Generate an intent vector containing spatial parameters based on the LLM inference unit. The intent vector = {action subject, action type, spatial parameters, constraints}. For example: the user says "go to the kitchen to get an apple" → which is parsed as: intent vector = {subject: robotic arm, type: grasping, space: kitchen, condition: avoid people}.

[0055] S4: Read the intent vector through the state manager and update the environment matrix in real time. The state manager maintains the environment matrix in real time. The environment matrix = {robot position, task queue, sensor status, environment map, interaction history}. The environment matrix is ​​automatically refreshed by subscribing to environmental sensor data.

[0056] S5: The task weight W is dynamically calculated using the updated environment matrix, and the priority of each task is dynamically adjusted. The formula for calculating the task weight W is: W = α * User urgency + β * Security level + γ * Energy consumption coefficient + δ * Module dependency. Where α, β, γ, and δ are weight coefficients, and these coefficients are constants ranging from 0% to 100%. The higher the task weight W, the higher the task priority, and the earlier the system processes it. The correspondence between the task weight W and the environment matrix is ​​as follows:

[0057]

[0058]

[0059] In application scenarios, the weighting coefficients α, β, γ, and δ in the formula are dynamically adjusted, as shown in the following example:

[0060] Assumption 1: In autonomous driving or service robot systems, during normal driving, tasks such as "path planning" and "lighting adjustment" may be being handled, at which point obstacle avoidance has a medium priority. However, when the sensors detect an obstacle ahead (such as a pedestrian or a wall), the obstacle avoidance task is directly associated with "avoiding collision (safety risk)". Therefore, the value of the "safety level" will be pulled from "low" to "extremely high". The safety coefficient β will be temporarily increased, and the system will instantly increase β (the weight coefficient of the safety level) to maximize the contribution of the "safety level" to the task weight W.

[0061] At this point, the task weight W of the obstacle avoidance task is α*(user urgency remains unchanged) + β(increases) (safety level increases) + γ(energy consumption coefficient is secondary) + δ*(module dependency is secondary), which causes the task weight W to increase significantly. The obstacle avoidance task jumps from "medium priority" to "highest priority". The system immediately suspends other non-critical tasks (such as adjusting the air conditioner) and prioritizes obstacle avoidance (braking, steering).

[0062] Hypothesis 2: When applied to a smart home control system, while handling tasks such as "playing music" and "recording energy consumption," the user's urgency level is maximized when the user issues an emergency call command for "help." The weight coefficient of "user urgency" is directly set to the "maximum value." The urgency coefficient α is adjusted to the maximum, and the system temporarily sets α (the weight coefficient of user urgency) to the highest value, even temporarily disabling the influence of other coefficients (β, γ, δ). This is equivalent to the "emergency call" triggering the "priority" rule.

[0063] At this point, the task weight W of the emergency task directly reaches the "maximum threshold" set by the system. The system immediately suspends other non-critical tasks and prioritizes tasks such as automatically dialing emergency contacts and triggering alarms in the security system.

[0064] S6: The corresponding functional modules are called according to the priority of each task through a plug-in interface. The called functional modules execute the corresponding tasks and send the task execution progress and results back to the state manager. Natural language feedback is generated and sent to the user through a text-to-speech (TTS) module, such as "Arrived at the kitchen, retrieving apples." The functional modules include: a navigation module, a robotic arm control module, and a vision tracking module. In practical applications, a collaborative arbitration mechanism is configured between the functional modules. This mechanism allocates corresponding hardware permissions to the corresponding functional modules based on the task weight W. For example, when the navigation module and the vision tracking module conflict, camera permissions are allocated according to the task weight W.

[0065] S7: Monitors the environmental matrix in real time. If an anomaly is detected, the current task is interrupted and an active voice interaction is triggered to notify the user of the abnormal situation and wait for further instructions. Anomalies include fire, falls, and obstructions. If no anomalies are detected, the process continues in a loop.

[0066] Example 2

[0067] like Figure 2 As shown, based on Embodiment 1, this embodiment provides a robot dynamic voice interaction system based on multimodal state management. The system includes:

[0068] A multimodal input module is provided to receive multimodal command data input by the user. The multimodal command data includes voice, text, and environmental sensor data. The multimodal input module supports multi-channel input such as speech recognition (ASR) and gesture recognition. The speech recognition (ASR) module integrates a noise reduction model to adapt to noisy environments. The environmental sensor data is acquired by environmental sensors, which include an image acquisition device, a lidar, a temperature and humidity sensor, and an infrared detector.

[0069] An intent parsing module is communicatively connected to a multimodal input module, the intent parsing module being used to generate an intent vector containing spatial parameters;

[0070] A state manager that communicates with the intent parsing module is used to dynamically update the environment matrix, updating the data of the environment matrix every 200ms;

[0071] The intent dispatcher, which communicates with the state manager, mainly performs task scheduling. It calculates task priority weights based on the environment matrix and calls the corresponding functional plugins according to the priority of the task weight W. At the same time, it arbitrates resource allocation according to priority weights. That is, when different tasks need to use the same hardware resources, the hardware resources are allocated to the corresponding tasks according to the task weight W. For example, when the navigation module and visual tracking conflict, the camera usage permission is allocated to the corresponding task according to the size of the task weight W.

[0072] A pluggable function library that communicates with the intent dispatcher includes multiple independently callable function plugins. Each function plugin is called to perform a corresponding task. The function plugins are accessed through a pluggable interface and support hot-swapping. For example, a "fall detection for the elderly" module can be added according to the application scenario.

[0073] The intent distributor and the state manager have a bidirectional communication connection, and they form a dual-central collaborative architecture through real-time data exchange, creating a closed-loop control system. The intent distributor monitors the environment matrix of the state manager in real time. When the intent distributor detects an anomaly in the environment matrix, it actively interrupts the current task and triggers active voice interaction. Environmental anomalies include, but are not limited to: obstacles obstructing the view, fire alarms, falls, communication loss, and power anomalies.

[0074] Compared with the traditional single-central system, the dual-central collaborative architecture has significant advantages: ① response latency is reduced from 500ms to 120ms; ② multi-task conflict rate is reduced by 62%; ③ anomaly recovery speed is increased by 3 times.

[0075] Example 3

[0076] Based on the robot dynamic voice interaction method and system based on multimodal state management proposed in Embodiments 1 and 2 above, this embodiment takes a smart home control scenario as an example, as follows:

[0077] Users want to control the robot vacuum cleaner to clean specific spaces (such as the living room) via voice commands. At the same time, for safety and comfort, they require the robot vacuum cleaner to avoid areas where children are active during cleaning, to ensure children's safety and avoid disturbing them.

[0078] S1 Voice Input: Users can give the command by voice: "Clean the living room but avoid the children."

[0079] S2 Intent Resolution (NLU): Parses the text and extracts the command intent: {Action subject: robot vacuum cleaner; Action type: cleaning; Spatial parameter: living room; Constraint: avoid people}

[0080] The structured instruction is generated as follows: {Action subject: robot vacuum cleaner, Action type: cleaning, Spatial parameter: living room, Constraint condition: avoid people}.

[0081] S3 Environmental Awareness: Utilizes cameras, infrared sensors, pressure sensors, and other sensors to detect the specific location and activity area of ​​children in the living room in real time, generating a dynamic environmental map of the children's area.

[0082] S4 State Manager updates the environment matrix in real time.

[0083] Based on the spatial parameters and constraints in the intention, and combined with the location information provided by the child positioning module, the area that needs to be avoided is identified; the current status of the robot vacuum and the child's location status are dynamically maintained, and the obstacle avoidance path is dynamically adjusted according to the child.

[0084] S5 performs task scheduling via the intent dispatcher.

[0085] S501: The "path planning module" is invoked to run, which combines the child obstacle avoidance zone to plan a reasonable and efficient cleaning path for the robot vacuum cleaner, and dynamically adjusts the path to avoid entering the children's activity area;

[0086] S501: Sends specific paths and control commands to the robot vacuum cleaner to start the cleaning task.

[0087] S502: The robot announces via voice: "Children's area avoided, cleaning the living room has begun"; it also provides real-time status feedback to the user, such as "Child detected approaching, path adjusted".

[0088] S6 exception handling

[0089] If a child gets too close to the robot's working area, the robot will automatically pause and prompt the user to initiate active voice interaction, triggering a safety mechanism through the status manager.

[0090] Example 4

[0091] Based on the robot dynamic voice interaction method and system based on multimodal state management proposed in Embodiments 1 and 2 above, this embodiment takes the control scenario of an industrial handling robot avoiding high-temperature areas as an example, as follows:

[0092] In industrial environments, users use voice commands to instruct handling robots to perform handling tasks, while also requiring the robots to avoid dangerous high-temperature areas to ensure the safety of equipment and personnel. When the high-temperature area changes dynamically, the system automatically adjusts the handling destination to ensure the safe completion of the task.

[0093] S1 Voice Input

[0094] The user issued the instruction: "Move the box to area A, and be careful to avoid the high-temperature area."

[0095] S2 Intent Resolution (NLU)

[0096] Parse the instructions, extract the intent, and form a structured instruction {Action subject: transport robot, Action type: transport box, Spatial parameter: area A, Constraint condition: avoid high temperature area}.

[0097] S3 Environmental Perception

[0098] By using thermal sensors, infrared cameras, and other technologies, the temperature of various areas within an industrial site can be monitored in real time to identify the current range and dynamic expansion of high-temperature zones.

[0099] S4 State Manager

[0100] Monitor the dynamic status of the high-temperature zone and update the environmental matrix in real time. Determine whether the target space A area is in a high-temperature risk zone. If high temperature is detected in A area, notify the intent distributor to adjust the task.

[0101] S4 Intent Distributor

[0102] Based on feedback from the status manager, the task parameters are automatically modified: the target space is changed from area A to the safe area B (or a preset backup area), the path planning module is invoked to combine the high-temperature area information to plan a safe handling path, avoid passing through high-temperature danger areas, and the handling path is dynamically adjusted to ensure safety.

[0103] The S5 intent dispatcher invokes functional plugins to execute actions.

[0104] New target and path instructions are issued to the handling robot, and the corresponding functional modules are invoked to execute the handling task.

[0105] S6 Feedback Interaction

[0106] The robot provides real-time feedback to the user via voice: "High temperature detected in area A, has been moved to area B."

[0107] S7 exception handling

[0108] If the high-temperature zone expands further or other abnormalities occur, the robot will pause its actions and initiate proactive voice interaction, waiting for human confirmation or automatically replanning the task.

[0109] It should be noted that any process or method description in the flowchart or otherwise described herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including in substantially the same manner or in the reverse order of the functions involved, as should be understood by those skilled in the art to which the embodiments of this application pertain.

[0110] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0111] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0112] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0113] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0114] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A robot dynamic voice interaction method based on multimodal state management, characterized in that, The method includes: S1: Collect multimodal command data input by the user; S2: After processing the multimodal instruction data, generate an input text package with environment context markers; S3: After parsing the intent of the input text packet, bind it with the historical context to generate an intent vector; S4: Update the environment matrix in real time using intent vectors; S5: Calculate the task weight W dynamically using the updated environment matrix, and dynamically adjust the priority of each task; S6: Through the plug-in interface, the corresponding functional modules are called according to the priority of each task, and the corresponding tasks are executed.

2. The robot dynamic voice interaction method based on multimodal state management according to claim 1, characterized in that, The multimodal command data includes: user voice commands, text information, and environmental sensor data, and the environmental sensor data includes: image acquisition data, lidar data, temperature and humidity data, and infrared detection data.

3. The robot dynamic voice interaction method based on multimodal state management according to claim 1, characterized in that, The data processing method for the multimodal command data is as follows: S201: The voice command is recognized and converted by the ASR module to generate structured text; S202: The environmental sensor data is timestamped and aligned with the structured text through the data fusion module; S203: Generate an input text package with environment context tags.

4. The robot dynamic voice interaction method based on multimodal state management according to claim 1, characterized in that, The method for generating the intent vector is as follows: S301: The natural language processing unit performs semantic understanding on the input text packet to generate a preliminary intent vector; S302: Bind the initial intent vector to the historical context of the state manager; S303: Generate an intent vector containing spatial parameters.

5. The robot dynamic voice interaction method based on multimodal state management according to claim 4, characterized in that, The intent vector is read through the state manager, and the environment matrix is ​​updated in real time.

6. The robot dynamic voice interaction method based on multimodal state management according to claim 1, characterized in that, The formula for calculating the task weight W is: W = α * User urgency + β * Security level + γ * Energy consumption coefficient + δ * Module dependency; where α, β, γ and δ are constants.

7. The robot dynamic voice interaction method based on multimodal state management according to claim 1, characterized in that, In step S6, the task execution progress and results are sent back to the state manager, and natural language is generated and fed back to the user through the speech synthesis module.

8. The robot dynamic voice interaction method based on multimodal state management according to claim 1, characterized in that, The method also includes: real-time monitoring of the environmental matrix; if an anomaly is detected, the current task is interrupted and active voice interaction is triggered; if no anomaly is detected, the process continues in a loop.

9. A robot dynamic voice interaction system based on multimodal state management, characterized in that, The system operates the robot dynamic voice interaction method based on multimodal state management as described in any one of claims 1-8, and the system includes: A multimodal input module, wherein the multimodal input module is used to receive multimodal command data input by the user; An intent parsing module is communicatively connected to a multimodal input module, the intent parsing module being used to generate an intent vector containing spatial parameters; A state manager that communicates with the intent parsing module, the state manager being used to dynamically update the environment matrix; An intent dispatcher that communicates with a state manager calculates task priority weights based on an environment matrix and arbitrates resource allocation for tasks according to the priority weights; A pluggable function library that communicates with the intent dispatcher includes multiple independently callable function plugins, each of which is used to perform a corresponding task.

10. The robot dynamic voice interaction system based on multimodal state management according to claim 9, characterized in that, The intent distributor and the state manager form a dual-central collaborative architecture through real-time data exchange.

Images