A natural language-based robot control method, device, equipment and medium

Abstract

The application provides a natural language-based robot control method and device, equipment and medium, comprising: performing audio signal conversion processing on a natural voice instruction sent by a user terminal to determine a text instruction of the natural voice instruction; performing multi-dimensional analysis processing on the text instruction based on a semantic understanding model, and performing structured processing on the multi-dimensional analysis result to determine a structured task instruction; and sending the structured task instruction to a main controller of a robot through a wireless communication network, so that after the main controller receives the structured task instruction, the main controller drives the robot main body to move to a target pose, controls an end effector to perform based on action parameters and a task graph, and feeds back an execution state to the user terminal in real time. The operator can directly and flexibly control the robot to complete the corresponding task through the spoken instruction, and the human-computer interaction efficiency and operation safety are improved.

Description

Technical Field

[0001] This application relates to the technical field of intelligent robot control, and in particular to a robot control method, apparatus, device and medium based on natural language. Background Technology

[0002] In the field of laboratory automation, especially in high-throughput sample processing, multi-step biochemical reactions, and aseptic operations, frequently opening and closing the lids of various containers (such as 1.5mL centrifuge tubes, 15mL / 50mL conical tubes, standard reagent bottles, etc.) is a basic but crucial operation. Currently, the mainstream technical solutions include the following two types: (1) Manual operation: Experimenters directly use their fingers or general tools (such as cap openers) to complete the capping action. Although this method is simple and flexible, it has the problem of low efficiency. (2) Pre-programmed robotic arm system: An industrial / collaborative robotic arm is equipped with a customized gripper, and a fixed action sequence is generated through teaching programming or offline trajectory planning (such as: "move to (x0,y0,z0) → clamping force F → rotate clockwise N times → release"). Although such systems can replace manual labor, they face the problems of strong environmental rigidity and serious disconnection of interactive logic in actual scientific research applications. Therefore, how to flexibly control the robot to complete the corresponding tasks has become a technical problem that cannot be ignored. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a robot control method, device, equipment and medium based on natural language, which enables real-time and precise robot control by using the spatial relationships, action intentions and physical execution parameters contained in natural voice commands, so that operators can directly and flexibly control the robot to complete corresponding tasks through spoken commands, thereby improving human-computer interaction efficiency and operational safety.

[0004] This application provides a natural language-based robot control method, which includes: The natural speech command sent by the user terminal is processed by audio signal conversion to determine the text command of the natural speech command; The text instructions are parsed in multiple dimensions based on a semantic understanding model, and the parsing results are then structured to determine the structured task instructions. The structured task instructions include the target pose, action type, action parameters, and task graph. The structured task instructions are sent to the robot's main controller via a wireless communication network. After receiving the structured task instructions, the main controller drives the robot body to move to the target pose, controls the end effector to execute based on the motion parameters and the task map, and feeds back the execution status to the user terminal in real time.

[0005] In one possible implementation, where the dimension is a spatial object positioning dimension, the multi-dimensional parsing processing of the text instruction based on the semantic understanding model includes: Dependency parsing is used to identify noun phrases as operation objects in text instructions; When the noun phrase is a spatial modifier, the locative words and ordinal numbers are extracted from it. Based on the environmental semantic map, the locative words are mapped to the axial definition of the robot coordinate system, and the ordinal numbers are mapped to the distance sorting rules along the axial direction to determine the target pose with physical meaning. The target pose includes at least the three-dimensional position coordinates of the center of the container opening.

[0006] In one possible implementation, for the dimension being the operation type positioning dimension, the multi-dimensional parsing processing of the text instruction based on the semantic understanding model includes: Based on the verbs and their contextual modifications in the text instructions, the operation intentions are classified into discrete rotation action types or continuous incremental rotation action types; wherein, the discrete rotation action type corresponds to a preset fixed rotation direction and fixed angle value, and the continuous incremental rotation action type corresponds to a closed-loop control mode based on real-time physical feedback.

[0007] In one possible implementation, for the dimension being a parametric semantic quantization dimension, the multi-dimensional parsing processing of the text instruction based on the semantic understanding model includes: Based on a preset quantization mapping rule base, the numerical parameters or fuzzy quantifier parameters contained in the text instructions are subjected to domain-adapted numerical mapping to determine the corresponding action parameters; wherein, the action parameters include rotation angle, angular velocity, force control threshold or termination criterion.

[0008] In one possible implementation, where the dimension is a temporal logical dimension, the multi-dimensional parsing processing of the text instruction based on the semantic understanding model includes: Identify temporal connectors or conditional adverbs in the text instructions, and construct a task graph based on the identification results; wherein, each node in the task graph corresponds to an execution action, and directed edges represent the execution relationships between nodes.

[0009] In one possible implementation, after the execution status is fed back to the user terminal in real time, the robot control method further includes: The type of the execution state is parsed, and multi-source heterogeneous context data associated with the type is dynamically retrieved. The multi-source heterogeneous context data and the type of execution state are input into the state-driven decision model to determine the human-machine collaborative processing strategy.

[0010] In one possible implementation, the step of inputting the multi-source heterogeneous context data and the type of execution state into a state-driven decision model to determine a human-machine collaborative processing strategy includes: When the execution status is a success state, an archiving action is triggered, which encapsulates the natural voice command, the structured task command, key sensor timing data, operator identification and timestamp into an encrypted audit unit and stores it persistently. When the execution status is a failure status, based on the joint analysis of the operator's historical operation behavior profile and the current key sensor time series data, the operator is provided with a parameter suggestion range that matches the current operation object.

[0011] This application also provides a robot control device based on natural language, the robot control device comprising: The speech recognition module is used to convert the natural speech commands sent by the user terminal into audio signals and determine the text commands of the natural speech commands. The semantic understanding module is used to perform multi-dimensional parsing of the text instructions based on the semantic understanding model, and to perform structuring processing on the multi-dimensional parsing results to determine the structured task instructions; wherein, the structured task instructions include target pose, action type, action parameters and task graph; The motion execution module is used to send the structured task instructions to the robot's main controller via a wireless communication network, so that after receiving the structured task instructions, the main controller drives the robot body to move to the target pose, controls the end effector to execute based on the motion parameters and the task map, and feeds back the execution status to the user terminal in real time.

[0012] This application also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the natural language-based robot control method described above are performed.

[0013] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the natural language-based robot control method described above.

[0014] This application provides a robot control method, apparatus, device, and medium based on natural language. The robot control method includes: converting natural speech commands sent by a user terminal into audio signals to determine the text commands; performing multi-dimensional parsing of the text commands based on a semantic understanding model, and performing structured processing on the multi-dimensional parsing results to determine structured task commands; wherein the structured task commands include target pose, action type, action parameters, and task graph; sending the structured task commands to the robot's main controller via a wireless communication network, so that after receiving the structured task commands, the main controller drives the robot body to move to the target pose, controls the end effector to execute based on the action parameters and the task graph, and feeds back the execution status to the user terminal in real time. By using the spatial relationships, action intentions, and physical execution parameters contained in natural speech commands for real-time and precise robot control, operators can directly and flexibly control the robot to complete corresponding tasks through spoken commands, improving human-computer interaction efficiency and operational safety.

[0015] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a natural language-based robot control method provided in this application embodiment; Figure 2 One of the structural schematic diagrams of a robot control device based on natural language provided in the embodiments of this application; Figure 3 A second schematic diagram of a robot control device based on natural language provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0019] First, the applicable application scenarios of this application will be introduced. This application can be applied to the technical field of intelligent robot control.

[0020] Research has revealed that the current mainstream technical solutions include the following two types: (1) Manual operation: Experimenters directly use their fingers or general tools (such as cap openers) to complete the cap-tightening action. Although this method is simple and flexible, it suffers from low efficiency. (2) Pre-programmed robotic arm system: An industrial / collaborative robotic arm is equipped with a customized gripper, and a fixed action sequence is generated through teaching programming or offline trajectory planning (e.g., "move to (x0,y0,z0) → clamping force F → rotate clockwise N times → release"). Although this type of system can replace manual labor, it faces the problems of strong environmental rigidity and serious disconnection of interaction logic in actual scientific research applications. Therefore, how to flexibly control the robot to complete the corresponding task has become a technical problem that cannot be ignored.

[0021] Based on this, this application provides a robot control method based on natural language. By using the spatial relationships, action intentions and physical execution parameters contained in natural speech commands, the robot can be controlled in real time and with precision. This allows operators to directly and flexibly control the robot to complete corresponding tasks through spoken commands, thereby improving human-computer interaction efficiency and operational safety.

[0022] Please see Figure 1 , Figure 1 This is a flowchart illustrating a natural language-based robot control method provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the robot control method includes: S101: Perform audio signal conversion processing on the natural speech command sent by the user terminal to determine the text command of the natural speech command.

[0023] In this step, the natural speech commands sent by the user are converted into audio signals to determine the text commands.

[0024] Here, natural speech commands sent by the user are collected using a dual-microphone array. The collected raw audio stream is then processed by a digital signal processor to perform adaptive noise suppression, echo cancellation, and speech activity detection to obtain clean natural speech commands.

[0025] The speech recognition module utilizes a lightweight edge-side ASR engine, which has been fine-tuned using laboratory scenario corpora: (a) Training corpus composition: It includes multiple recordings of real researchers issuing cap-tightening instructions (covering dialects from both the north and south, variations in speech rate, and slight accents), encompassing typical instruction paradigms such as "tighten the middle one in the third row," "loosen the blue tube cap by half a turn," and "tighten bottle A first, then loosen bottle B," etc.; (b) Acoustic modeling enhancement: Based on MFCC features, additional fundamental frequency contour and energy envelope dual-channel features are introduced to improve the robustness of pronunciation for key verbs such as "tighten," "loosen," and "tighten"; (c) Language model constraints: A domain-restricted language model is embedded, forcing the decoder to prioritize outputting word combinations that conform to the grammar of the cap-tightening task (e.g., prohibiting invalid collocations such as "tighten the refrigerator" and "tighten the mouse").

[0026] Here, after receiving the text instruction, the syntax is validated. For example, the regular expression engine is called to match the preset instruction template (such as "[action verb][operation object][quantification parameter]"). For text that is missing key elements (such as no action verb or no object description), a voice prompt is triggered: "Please say it again, for example, 'Unscrew the first test tube on the left'."

[0027] S102: The text instructions are parsed in multiple dimensions based on a semantic understanding model, and the parsing results are structured to determine the structured task instructions; wherein, the structured task instructions include target pose, action type, action parameters, and task graph.

[0028] In this step, the text instructions are parsed in multiple dimensions based on the semantic understanding model, including spatial object location dimension, operation type location dimension, parameter semantic quantification dimension, and temporal logic dimension, to obtain multi-dimensional results. The multi-dimensional parsing results are then structured to determine the structured task instructions.

[0029] The structured task instructions include the target pose, action type, action parameters, and task graph. It should be noted that the task graph represents the robot's task execution sequence.

[0030] It should be noted that the semantic understanding model is the Natural Language Understanding model.

[0031] In one possible implementation, where the dimension is a spatial object positioning dimension, the multi-dimensional parsing processing of the text instruction based on the semantic understanding model includes: A: Dependency parsing is performed on the text instructions to identify noun phrases as the objects of operation.

[0032] Here, for a text instruction (e.g., "unscrew the cap of the first test tube on the left"), a lightweight dependency parser (such as a Chinese dependency parsing model based on BERT+CRF) is invoked to identify all noun phrases. Based on the dependency arc types in the dependency tree, such as `nsubj` (subject), `dobj` (direct object), and `attr` (attribute modifier), semantics with operable entities are filtered out. For example, this is identified as "the first test tube on the left," and its part-of-speech sequence is: `[left / locative noun] + [first / ordinal number] + [test tube / central noun]`.

[0033] B: When the noun phrase is a spatial modifier, extract the locative words and ordinal numbers from it, map the locative words to the axial definition of the robot coordinate system based on the environmental semantic map, and map the ordinal numbers to the distance sorting rules along the axial direction to determine the target pose with physical meaning; wherein, the target pose includes at least the three-dimensional position coordinates of the center of the container opening.

[0034] Here, directional terms are mapped to the physical axis definitions of the robot's base coordinate system ({X,Y,Z}). This mapping relationship is pre-defined and dynamically maintained by the environmental semantic map. The environmental semantic map is a lightweight 3D topology map that records the rigid body transformation matrices between fixed reference points on the experimental platform (such as the four corners of the platform and camera mounting positions) and the robot's base coordinate system, as well as the semantic labels of common container placement areas (such as "main reagent rack → positive Y-axis extension area" and "centrifuge tube array → XY plane grid area"). For example, when the "main reagent rack" is defined in the environmental semantic map as being located in the positive Y-axis direction of the robot, then "left" in the command is mapped to the negative X-axis direction, and "front" is mapped to the positive Y-axis direction. This mapping is not a static dictionary lookup, but rather a real-time pose calculation of reference points in the environmental semantic map, supporting adaptive updates after platform fine-tuning.

[0035] Specifically, for ordinal numbers, a one-dimensional projection sort is performed along the axial direction they modify within the corresponding semantic region of the environmental semantic map. For example: "first on the left" → projected onto the negative X-axis → extract the X coordinates of the center points of all container openings within the region → sorted in ascending order of X values ​​→ take the first one; "far right" → positive X-axis → take the one with the largest X coordinate. The sorting is based on the three-dimensional coordinates (x, y, z) of the center points of the container openings, where the z coordinate is compensated and calibrated by a preset container type height template in the environmental semantic map (e.g., "15mL centrifuge tube opening height = 120mm") to ensure accurate grasping pose.

[0036] In one possible implementation, for the dimension being the operation type positioning dimension, the multi-dimensional parsing processing of the text instruction based on the semantic understanding model includes: Based on the verbs and their contextual modifications in the text instructions, the operation intentions are classified into discrete rotation action types or continuous incremental rotation action types; wherein, the discrete rotation action type corresponds to a preset fixed rotation direction and fixed angle value, and the continuous incremental rotation action type corresponds to a closed-loop control mode based on real-time physical feedback.

[0037] Here, in order to ensure that the robot performs actions that strictly match the user's true intentions (avoiding "unscrew" being mistakenly executed as "rotate 360°" or "stop when loose"), this application proposes a binary classification mechanism for action types based on verb semantic role labeling and contextual modifier constraints. This mechanism accurately maps the operational intentions in natural language instructions to two physically distinguishable rotation control paradigms: discrete rotation action types and continuous incremental rotation action types.

[0038] It should be noted that verbs include main predicates: such as “tighten”, “tighten”, “loosen”, “turn”; manner adverbs: such as “slowly”, “with force”, “gently”, “until”; result complements: such as “open”, “tight”, “loose”, “click”; quantity complements: such as “half a circle”, “two circles”, “a little bit”.

[0039] The discrete rotation action type is determined as follows: the system looks up a table and calls the preset action template library. Each template includes: fixed rotation direction: determined by the polarity of the verb - "open" → counterclockwise, "tighten" → clockwise; fixed angle value: determined by the container type and cap specification (e.g., standard 15mL centrifuge tube screw cap: fully open = 360°, slightly loose = 45°); clamping torque: preset according to the container material (plastic cap: 0.3N·m; metal screw cap: 0.8N·m).

[0040] Here, the continuous incremental rotation motion type refers to a closed-loop control mode in which the robot's end effector continuously collects and responds to at least one real-time physical feedback signal (torque, acoustic, angular acceleration) during rotation, and determines whether to continue, decelerate, or stop the rotation based on a preset dynamic termination criterion. Its rotation endpoint cannot be predetermined and depends on the evolution of the physical interaction process itself. For example, rotational motion has an uncertain endpoint, and the control strategy is a force-position hybrid dual-loop PID: the outer loop plans the torque ramp curve, and the inner loop tracks the position command, allowing interruption at any position due to feedback triggering. For example, the semantic parsing module identifies commands that simultaneously contain continuous mode adverbs (such as "slowly," "gradually," "carefully") and process-oriented result complements (such as "until it loosens," "until there is resistance," "until it feels lighter").

[0041] In one possible implementation, for the dimension being a parametric semantic quantization dimension, the multi-dimensional parsing processing of the text instruction based on the semantic understanding model includes: Based on a preset quantization mapping rule base, the numerical parameters or fuzzy quantifier parameters contained in the text instructions are subjected to domain-adapted numerical mapping to determine the corresponding action parameters; wherein, the action parameters include rotation angle, angular velocity, force control threshold or termination criterion.

[0042] Here, to transform unstructured parameter expressions in text instructions (such as "twist half a turn", "relax slightly", "tighten forcefully") into engineering parameters that the robot's main controller can parse, this invention constructs and runs a preset quantization mapping rule base. The mapping process of this rule base strictly follows a three-level adaptation logic of "semantic category → domain context → physical parameters," ensuring that action parameters possess repeatability, safety, and biological / chemical experimental specificity. For example, "half a turn" → the quantifier-value mapping table in the quantization mapping rule base returns 180 degrees.

[0043] In one possible implementation, for the dimension being a temporal logic dimension, the multi-dimensional parsing processing of the text instruction based on the semantic understanding model includes: identifying temporal connectors or conditional adverbs in the text instruction, and constructing a task graph based on the identification results; wherein, each node in the task graph corresponds to an execution action, and directed edges represent the execution relationship between nodes.

[0044] Here, temporal connectors or conditional adverbs in the text instructions are identified. Temporal connectors include "first…then…", "then…", "next…", "except…", and "simultaneously…", while conditional adverbs include "if…then…" and "when…". A task graph is constructed based on the identification results. Each node in the task graph corresponds to a cap-tightening action, and node attributes include execution priority, clamping force precondition, and torque triggering condition. Directed edges represent the execution relationships between nodes, including sequential execution, parallel execution, conditional triggering, or failure rollback relationships. The output task graph serves as the temporal logic layer of the structured task instructions, providing the robot's main controller with motion planning and closed-loop execution.

[0045] S103: The structured task instruction is sent to the robot's main controller via a wireless communication network, so that after receiving the structured task instruction, the main controller drives the robot body to move to the target pose, controls the end effector to execute based on the motion parameters and the task map, and feeds back the execution status to the user terminal in real time.

[0046] In this step, structured task instructions are sent to the robot's main controller via a wireless communication network. After receiving the structured task instructions, the main controller drives the robot body to move to the target pose, controls the end effector to execute according to the motion parameters and task map, and feeds back the execution status to the user terminal in real time.

[0047] Here, after receiving the task, the robot's main controller performs motion planning and sequentially executes the control of the robotic arm positioning, controls the electric rotating gripper to hold the cover, controls it to rotate in the specified direction and angle, and controls it to release and move away.

[0048] In one possible implementation, after the execution status is fed back to the user terminal in real time, the robot control method further includes: (1): Analyze the type of the execution state and dynamically retrieve the associated multi-source heterogeneous context data based on the type.

[0049] Here, multi-source heterogeneous context data associated with it is dynamically retrieved. The multi-source heterogeneous context data includes at least: (i) the original natural language instruction text of the task corresponding to the execution state; (ii) key sensor timing data that characterizes the physical behavior of the task execution process and is synchronously uploaded by the robot's main controller. The key sensor timing data includes at least the clamping force signal, the output torque signal, the rotation angle feedback signal, and the time consumption information of each control sub-step; (iii) the operator's identity and historical operation behavior profile. The historical operation behavior profile consists of statistical information on the operation results of the operator performing similar tasks within a preset time window. The statistical information includes the task success rate, the frequency distribution of abnormal events, and the physical specifications of high-frequency operation objects.

[0050] (2): Input the multi-source heterogeneous context data and the type of execution state into the state-driven decision model to determine the human-machine collaborative processing strategy.

[0051] Here, the state-driven decision model takes structured multi-source heterogeneous context data and the robot's real-time execution state as input, and outputs a human-machine collaborative processing strategy with clear operational semantics.

[0052] In one possible implementation, the step of inputting the multi-source heterogeneous context data and the type of execution state into a state-driven decision model to determine a human-machine collaborative processing strategy includes: a: When the execution status is a success state, an archiving action is triggered, which encapsulates the natural voice command, the structured task command, key sensor timing data, operator identity identifier and timestamp into an encrypted audit unit and stores it persistently.

[0053] b: When the execution status is a failure status, based on the joint analysis of the operator's historical operation behavior profile and the current key sensor time series data, the operator is provided with a parameter suggestion range that matches the current operation object.

[0054] Here, when the execution status is a failure state, based on the joint analysis of the operator's historical operation behavior profile and the current key sensor timing data, the operator is provided with a parameter suggestion range that matches the current operation object, so that the control robot can re-execute the operation object according to the parameter suggestion range.

[0055] In a specific embodiment, Step 1: Receiving Voice Commands: The process begins with the system receiving natural language voice commands from the user via the computer's voice acquisition module. Step 2: Automatic Speech Recognition (ASR). The system calls the speech recognition engine to convert the audio signal into corresponding text commands. Step 3: Natural Language Understanding (NLU) and Task Generation: The natural language understanding module parses the text commands, identifying: the object of operation (e.g., "the first test tube on the left," and converting it into target coordinates), the operation action (e.g., "unscrew" mapped to rotation - counterclockwise), and the action parameters (e.g., "half a circle" parsed as...). (180 degrees). Finally, structured robot task instructions are generated, for example: {Target position: (x1, y1, z1), Action sequence: [Clamping (force F), Rotation (counterclockwise, angle A), Release]}. Step 4: Wireless task transmission: The structured task instructions are sent to the robot's main controller via a wireless network. Step 5: Motion planning and execution: After receiving the task, the robot's main controller performs motion planning and executes the following sequentially: a) Control the robotic arm to position; b) Control the electric rotating gripper to clamp the cover; c) Control its rotation in the specified direction and angle; d) Release and remove. Step 6: Feedback on execution results: The robot feeds back the task execution status (success / failure / abnormality) to the user.

[0056] Existing pre-programmed capping systems require offline calibration and hard coding based on container model and spatial pose, necessitating manual recalibration or reprogramming for each environmental change (such as test tube replacement or sample rack relocation). This application addresses this by using a natural language semantic understanding module to parse spatial descriptive instructions (such as "first one on the left" or "top blue tube") in real time, mapping fuzzy language to target positions in the robot coordinate system, and linking it with the motion planning module to complete autonomous localization and execution—without modifying the underlying control program or recalibrating the vision system, significantly improving the system's robustness and deployment efficiency in highly dynamic scientific research environments.

[0057] This application provides a natural language-based robot control method. The method includes: converting natural language commands sent from a user terminal into audio signals to determine the text commands; performing multi-dimensional parsing of the text commands based on a semantic understanding model; and structuring the parsing results to determine structured task commands. The structured task commands include a target pose, action type, action parameters, and a task graph. The structured task commands are then sent to the robot's main controller via a wireless communication network. Upon receiving the structured task commands, the main controller drives the robot body to the target pose, controls the end effector to execute actions based on the action parameters and the task graph, and provides real-time feedback of the execution status to the user terminal. By enabling real-time and precise robot control through the spatial relationships, action intentions, and physical execution parameters contained in natural language commands, operators can directly and flexibly control the robot to complete corresponding tasks using spoken commands, improving human-computer interaction efficiency and operational safety.

[0058] Please see Figure 2 , Figure 3 , Figure 2 One of the structural schematic diagrams of a robot control device based on natural language provided in the embodiments of this application; Figure 3 This is a second schematic diagram of a robot control device based on natural language, provided as an embodiment of this application. Figure 2 As shown, the natural language-based robot control device 200 includes: The speech recognition module 210 is used to perform audio signal conversion processing on the natural speech commands sent by the user terminal and determine the text commands of the natural speech commands. The semantic understanding module 220 is used to perform multi-dimensional parsing processing on the text instructions based on the semantic understanding model, and to perform structuring processing on the multi-dimensional parsing results to determine the structured task instructions; wherein, the structured task instructions include target pose, action type, action parameters and task graph; The motion execution module 230 is used to send the structured task instruction to the robot's main controller via a wireless communication network, so that after receiving the structured task instruction, the main controller drives the robot body to move to the target pose, controls the end effector to execute based on the motion parameters and the task map, and feeds back the execution status to the user terminal in real time.

[0059] Furthermore, the semantic understanding module 220 is used to locate the spatial object based on the dimension, and the text instruction is parsed and processed in multiple dimensions based on the semantic understanding model: Dependency parsing is used to identify noun phrases as operation objects in text instructions; When the noun phrase is a spatial modifier, the locative words and ordinal numbers are extracted from it. Based on the environmental semantic map, the locative words are mapped to the axial definition of the robot coordinate system, and the ordinal numbers are mapped to the distance sorting rules along the axial direction to determine the target pose with physical meaning. The target pose includes at least the three-dimensional position coordinates of the center of the container opening.

[0060] Furthermore, the semantic understanding module 220 is used to locate the dimension for the operation type, and the text instruction is parsed and processed in multiple dimensions based on the semantic understanding model: Based on the verbs and their contextual modifications in the text instructions, the operation intentions are classified into discrete rotation action types or continuous incremental rotation action types; wherein, the discrete rotation action type corresponds to a preset fixed rotation direction and fixed angle value, and the continuous incremental rotation action type corresponds to a closed-loop control mode based on real-time physical feedback.

[0061] Furthermore, the semantic understanding module 220 is used for multi-dimensional parsing of the text instruction based on the semantic understanding model, specifically for the dimension being a parameterized semantic quantization dimension. Based on a preset quantization mapping rule base, the numerical parameters or fuzzy quantifier parameters contained in the text instructions are subjected to domain-adapted numerical mapping to determine the corresponding action parameters; wherein, the action parameters include rotation angle, angular velocity, force control threshold or termination criterion.

[0062] Furthermore, the semantic understanding module 220 is used to perform multi-dimensional parsing processing on the text instructions based on the semantic understanding model, specifically for the temporal logical dimension: Identify temporal connectors or conditional adverbs in the text instructions, and construct a task graph based on the identification results; wherein, each node in the task graph corresponds to an execution action, and directed edges represent the execution relationships between nodes.

[0063] Furthermore, such as Figure 3 As shown, the natural language-based robot control device 200 also includes a policy generation module 240, which is used for: The type of the execution state is parsed, and multi-source heterogeneous context data associated with the type is dynamically retrieved. The multi-source heterogeneous context data and the type of execution state are input into the state-driven decision model to determine the human-machine collaborative processing strategy.

[0064] Furthermore, the strategy determination module 240 is used to input the multi-source heterogeneous context data and the type of execution state into the state-driven decision model to determine the human-machine collaborative processing strategy: When the execution status is a success state, an archiving action is triggered, which encapsulates the natural voice command, the structured task command, key sensor timing data, operator identification and timestamp into an encrypted audit unit and stores it persistently. When the execution status is a failure status, based on the joint analysis of the operator's historical operation behavior profile and the current key sensor time series data, the operator is provided with a parameter suggestion range that matches the current operation object.

[0065] This application provides a robot control device based on natural language. The robot control device includes: a speech recognition module for converting natural language commands sent by a user terminal into audio signals and determining the text commands; a semantic understanding module for performing multi-dimensional parsing of the text commands based on a semantic understanding model, and performing structured processing on the multi-dimensional parsing results to determine structured task commands; wherein the structured task commands include a target pose, action type, action parameters, and a task graph; and a motion execution module for sending the structured task commands to the robot's main controller via a wireless communication network, so that after receiving the structured task commands, the main controller drives the robot body to move to the target pose, controls the end effector to execute based on the action parameters and the task graph, and feeds back the execution status to the user terminal in real time. By using the spatial relationships, action intentions, and physical execution parameters contained in natural language commands for real-time and precise robot control, operators can directly and flexibly control the robot to complete corresponding tasks through spoken commands, improving human-computer interaction efficiency and operational safety. By enabling real-time and precise robot control through the spatial relationships, action intentions, and physical execution parameters contained in natural voice commands, operators can directly and flexibly control the robot to complete corresponding tasks through spoken commands, thereby improving human-computer interaction efficiency and operational safety.

[0066] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 400 includes a processor 410, a memory 420, and a bus 430.

[0067] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, they can perform the operations described above. Figure 1 The steps of the natural language-based robot control method in the illustrated method embodiment can be found in the method embodiment for specific implementation details, which will not be repeated here.

[0068] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of the natural language-based robot control method in the illustrated method embodiment can be found in the method embodiment for specific implementation details, which will not be repeated here.

[0069] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0070] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0071] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0072] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0073] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0074] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A robot control method based on natural language, characterized in that, The robot control method includes: The natural speech command sent by the user terminal is processed by audio signal conversion to determine the text command of the natural speech command; The text instructions are parsed in multiple dimensions based on a semantic understanding model, and the parsing results are then structured to determine the structured task instructions. The structured task instructions include the target pose, action type, action parameters, and task graph. The structured task instructions are sent to the robot's main controller via a wireless communication network. After receiving the structured task instructions, the main controller drives the robot body to move to the target pose, controls the end effector to execute based on the motion parameters and the task map, and feeds back the execution status to the user terminal in real time.

2. The robot control method according to claim 1, characterized in that, Regarding the dimension being the spatial object positioning dimension, the multi-dimensional parsing processing of the text instruction based on the semantic understanding model includes: Dependency parsing is used to identify noun phrases as operation objects in text instructions; When the noun phrase is a spatial modifier, the locative words and ordinal numbers are extracted from it. Based on the environmental semantic map, the locative words are mapped to the axial definition of the robot coordinate system, and the ordinal numbers are mapped to the distance sorting rules along the axial direction to determine the target pose with physical meaning. The target pose includes at least the three-dimensional position coordinates of the center of the container opening.

3. The robot control method according to claim 1, characterized in that, For the dimension being the operation type positioning dimension, the multi-dimensional parsing processing of the text instruction based on the semantic understanding model includes: Based on the verbs and their contextual modifications in the text instructions, the operation intentions are classified into discrete rotation action types or continuous incremental rotation action types; wherein, the discrete rotation action type corresponds to a preset fixed rotation direction and fixed angle value, and the continuous incremental rotation action type corresponds to a closed-loop control mode based on real-time physical feedback.

4. The robot control method according to claim 1, characterized in that, For the dimension being a parametric semantic quantization dimension, the multi-dimensional parsing processing of the text instruction based on the semantic understanding model includes: Based on a preset quantization mapping rule base, the numerical parameters or fuzzy quantifier parameters contained in the text instructions are subjected to domain-adapted numerical mapping to determine the corresponding action parameters; wherein, the action parameters include rotation angle, angular velocity, force control threshold or termination criterion.

5. The robot control method according to claim 1, characterized in that, For the aforementioned dimension being a temporal logical dimension, the multi-dimensional parsing processing of the text instructions based on the semantic understanding model includes: Identify temporal connectors or conditional adverbs in the text instructions, and construct a task graph based on the identification results; wherein, each node in the task graph corresponds to an execution action, and directed edges represent the execution relationships between nodes.

6. The robot control method according to claim 1, characterized in that, After the execution status is fed back to the user terminal in real time, the robot control method further includes: The type of the execution state is parsed, and multi-source heterogeneous context data associated with the type is dynamically retrieved. The multi-source heterogeneous context data and the type of execution state are input into the state-driven decision model to determine the human-machine collaborative processing strategy.

7. The robot control method according to claim 6, characterized in that, The step of inputting the multi-source heterogeneous context data and the type of execution state into the state-driven decision model to determine the human-machine collaborative processing strategy includes: When the execution status is a success state, an archiving action is triggered, which encapsulates the natural voice command, the structured task command, key sensor timing data, operator identification and timestamp into an encrypted audit unit and stores it persistently. When the execution status is a failure status, based on the joint analysis of the operator's historical operation behavior profile and the current key sensor time series data, the operator is provided with a parameter suggestion range that matches the current operation object.

8. A robot control device based on natural language, characterized in that, The robot control device includes: The speech recognition module is used to convert the natural speech commands sent by the user terminal into audio signals and determine the text commands of the natural speech commands. The semantic understanding module is used to perform multi-dimensional parsing of the text instructions based on the semantic understanding model, and to perform structuring processing on the multi-dimensional parsing results to determine the structured task instructions; wherein, the structured task instructions include target pose, action type, action parameters and task graph; The motion execution module is used to send the structured task instructions to the robot's main controller via a wireless communication network, so that after receiving the structured task instructions, the main controller drives the robot body to move to the target pose, controls the end effector to execute based on the motion parameters and the task map, and feeds back the execution status to the user terminal in real time.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the natural language-based robot control method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the natural language-based robot control method as described in any one of claims 1 to 7.

Images