Natural language driven mechanical arm control method, device, equipment, medium and product

Abstract

The invention discloses a natural language driven mechanical arm control method, device and equipment, a medium and a product, and relates to the technical field of artificial intelligence and robot crossing. The method comprises the steps that after a natural language instruction and multi-mode perception data are obtained, the natural language instruction is decomposed into semantic embedding vectors, large language model hierarchical semantic analysis is adopted to obtain structured task representation, and then the multi-mode perception data are fused to generate environment representation needed by task execution. And then the structured task representation and the environment representation are converted into a task graph, the task graph is converted into a joint control instruction of the target mechanical arm, and finally the joint control instruction is sent to a joint controller to be executed, so that through multi-level semantic understanding, multi-mode perception fusion and self-adaptive execution control, the joint control accuracy is improved. Accurate understanding and efficient execution of the mechanical arm on complex natural language instructions can be achieved, and the task execution precision, adaptability, flexibility, usability and intelligent level of the mechanical arm in a dynamic environment are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence and robot, and particularly relates to a natural language driven robot arm control method, device, equipment, medium and product. BACKGROUND

[0002] As the core execution unit in the field of industrial automation and service robots, the development of robot arm control technology is crucial to improving production efficiency and human-robot collaboration level.

[0003] At present, the existing robot arm control schemes mainly include programmed control scheme, teaching control scheme, vision-guided control scheme and strategy-based control scheme. However, these schemes have the following limitations in practical application: (1) Programmed control scheme: It is difficult to adapt to dynamic task requirements and is not user-friendly for non-technical users; (2) Teaching control scheme: It is tedious and inefficient, and cannot handle complex or variable tasks; (3) Vision-guided control scheme: It is sensitive to environmental changes and difficult to handle occlusion and illumination changes, and is usually limited to simple tasks; (4) Strategy-based control scheme: It lacks universality and is difficult to extend to new tasks or new environments.

[0004] With the advancement of artificial intelligence technology, especially the breakthrough of large language models in natural language processing, how to combine large language models, multi-modal perception and adaptive control technology to realize the intelligent understanding and execution of robot arms to natural language instructions, thereby significantly improving their flexibility, ease of use and intelligence level, is a topic that needs to be studied by those skilled in the art. SUMMARY

[0005] The purpose of the present application is to provide a natural language driven robot arm control method, device, control equipment, computer readable storage medium and computer program product, to solve the problem that the existing robot arm control scheme is limited in flexibility, ease of use and intelligence level.

[0006] In order to achieve the above-mentioned purpose, the present application adopts the following technical scheme: In a first aspect, a natural language driven robot arm control method is provided, which is executed by a control device, wherein the control device is respectively connected in communication with a language instruction input device, a multi-modal perception device and a joint controller of a target robot arm, and the multi-modal perception device is located in the surrounding environment of the target robot arm. The natural language driven robot arm control method comprises: acquiring a natural language instruction input by the language instruction input device and multi-modal perception data collected by the multi-modal perception device, wherein the multi-modal perception data includes visual data, tactile data and environmental data; The natural language instruction is decomposed into a semantic embedding vector, and a large language model based on a Transformer structure is used to perform hierarchical semantic parsing processing on the semantic embedding vector to obtain a structured task representation, wherein the hierarchical semantic parsing processing includes sequentially performed intent recognition layer semantic parsing processing, parameter extraction layer semantic parsing processing, and task combination layer semantic parsing processing, and the structured task representation includes a task intent obtained through the intent recognition layer semantic parsing processing, a task parameter obtained through the parameter extraction layer semantic parsing processing, and at least one identified subtask and an execution order of the at least one identified subtask obtained through the task combination layer semantic parsing processing; The multi-modal perception data are fused to generate an environment representation required for task execution; The structured task representation and the environment representation are converted into a task graph, wherein the nodes in the task graph represent the identified subtasks, and the edges in the task graph represent the execution order; The task graph is converted into joint control instructions of the target robot arm; The joint control instructions are sent to the joint controller for execution.

[0007] Based on the above invention content, a new scheme for natural language driven robot arm control combining large language model, multi-modal perception and adaptive control technology is provided, that is, after obtaining natural language instructions and multi-modal perception data, the natural language instructions are first decomposed into semantic embedding vectors, and a large language model is used to perform hierarchical semantic parsing to obtain a structured task representation, then the multi-modal perception data are fused to generate an environment representation required for task execution, then the structured task representation and the environment representation are converted into a task graph, and the task graph is converted into joint control instructions of the target robot arm, finally the joint control instructions are sent to the joint controller for execution, in this way, through multi-level semantic understanding, multi-modal perception fusion and adaptive execution control, the robot arm can accurately understand and efficiently execute complex natural language instructions, and the task execution precision, adaptability, flexibility, ease of use and intelligent level of the robot arm in a dynamic environment are improved, and the robot arm is particularly suitable for application scenarios such as intelligent manufacturing, service robots and medical assistance that require high-precision human-machine interaction, and is convenient for practical application and promotion.

[0008] In one possible design, in the process of the hierarchical semantic parsing processing, the loss function of semantic parsing is represented as follows:

[0009] wherein, represents the total number of natural language instruction samples, represents a number less than or equal to a positive integer, denotes a true semantic label corresponding to the th natural language instruction sample, denotes the th natural language instruction sample, denotes model parameters of the large language model, denotes a probability distribution function predicted by the large language model.

[0010] In one possible design, the multi-modal perception data is fused, including: a multi-modal dynamic fusion network based on a self-attention mechanism is adopted to optimize modal contribution of each modal perception data in the multi-modal perception data in real time according to task requirements and environmental changes, and then the multi-modal perception data is fused according to all the modal contributions, wherein the modal contribution adopts a fusion weight represented by the following formula:

[0011] wherein, denotes a total number of modes of the multi-modal perception data, and denote positive integers less than or equal to , denotes a fusion weight of the th modal perception data in the multi-modal perception data, denotes a natural exponential function, denotes a preset adjustment parameter, denotes a relevance of the th modal perception data to a target task and has , denotes a feature vector of the th modal perception data, denotes a transpose symbol, denotes a feature vector of the target task, that is, the structured task representation, denotes a confidence of the th modal perception data, denotes a Sigmoid function, denotes a sensitivity of the th modal perception data to environmental changes and has , denotes a preset environmental change sensitivity coefficient, denotes an environmental feature vector at a current moment, denotes an environmental feature vector at a previous moment, denotes the The correlation between each modal sensing data and the target task. Indicates the first The sensitivity of each modal sensing data to environmental changes.

[0012] In one possible design, the structured task representation and the environment representation are transformed into a task graph, including: A task graph generation model based on graph neural networks is used to transform the structured task representation and the environment representation into a task graph. In this task graph, nodes represent identified subtasks, and edges represent the execution order. The task graph generation model aims to minimize the optimization objective. It is expressed as follows:

[0013] In the formula, This represents the total number of natural language instruction samples. Indicates less than or equal to positive integers, Indicates the relationship with the first Structured task representations corresponding to each natural language instruction sample. Indicates the relationship with the first The environment representation corresponding to the nth multimodal sensing data sample, the nth The first natural language instruction sample and the first It was obtained by acquiring multiple multimodal sensing data samples together. Indicates the first The first natural language instruction sample and the first The real task graph labels corresponding to each multimodal sensing data sample. This represents the probability distribution function predicted by the task graph generation model. This represents the preset complexity penalty coefficient. Indicates less than or equal to positive integers, This indicates the number of nodes in the generated task graph. This represents the number of edges in the generated task graph, where the generated task graph refers to the graph generated by the task graph generation model that connects the first edge to the second edge. The structured task representation corresponding to the first natural language instruction sample and its relation to the first... The task graph is obtained by transforming the environmental representation corresponding to each multimodal sensing data sample.

[0014] In one possible design, the task diagram is translated into joint control commands for the target robotic arm, including: In the process of converting the task diagram into joint control commands for the target robotic arm, an inverse kinematics algorithm capable of adaptive adjustment with multiple constraints is used to solve the following optimization problem to obtain the joint angle vector used to generate the joint control commands:

[0015] In the formula, Indicates minimization. This represents the joint angle vector. Represents the target space coordinate vector. Represents the forward kinematic function. Represents positive integers. Represents the first in the multiple constraints The constraints are in functional form, and the multiple constraints include joint limit constraints, load constraints, and / or obstacle constraints. This represents the preset constraint weight coefficient.

[0016] In one possible design, after sending the joint control command to the joint controller for execution, the method further includes: Collect the execution status monitoring results of the target robotic arm; A reinforcement learning-based controller is used to optimize the execution of the target robotic arm's control tasks based on the execution state monitoring results, wherein the reward function of the reinforcement learning... It is expressed as follows:

[0017] In the formula, 、 and These represent the preset weighting coefficients. Denotes the base of the natural logarithm. This indicates the control accuracy of the target robotic arm. This indicates the speed at which the target robotic arm executes its control tasks. This indicates the number of errors in the execution of the control task of the target robotic arm.

[0018] In a second aspect, a natural language driven robotic arm control device is provided, which is suitable for being arranged in a control device, wherein the control device is communicatively connected to a language command input device, a multimodal sensing device and a joint controller of the target robotic arm, and the multimodal sensing device is located in the surrounding environment of the target robotic arm; The natural language driven robotic arm control device includes an instruction data acquisition unit, a semantic parsing and processing unit, a perception data fusion unit, a task graph conversion unit, a control instruction conversion unit, and a control instruction sending unit. The instruction data acquisition unit is configured to acquire natural language instructions input by the language instruction input device and multi-modal perception data collected by the multi-modal perception device, wherein the multi-modal perception data comprises visual data, tactile data and environmental data. The semantic analysis processing unit is communicatively connected to the instruction data acquisition unit and configured to decompose the natural language instructions into semantic embedding vectors and perform hierarchical semantic analysis processing on the semantic embedding vectors by using a large language model based on a Transformer structure to obtain a structured task representation, wherein the hierarchical semantic analysis processing comprises sequentially performed intent recognition layer semantic analysis processing, parameter extraction layer semantic analysis processing and task combination layer semantic analysis processing, and the structured task representation comprises a task intent obtained by the intent recognition layer semantic analysis processing, a task parameter obtained by the parameter extraction layer semantic analysis processing, at least one identified subtask obtained by the task combination layer semantic analysis processing and an execution order of the at least one identified subtask. The perception data fusion unit is communicatively connected to the instruction data acquisition unit and configured to fuse the multi-modal perception data to generate an environmental representation required for task execution. The task graph conversion unit is respectively communicatively connected to the semantic analysis processing unit and the perception data fusion unit and configured to convert the structured task representation and the environmental representation into a task graph, wherein a node in the task graph represents the identified subtask and an edge in the task graph represents the execution order. The control instruction conversion unit is communicatively connected to the task graph conversion unit and configured to convert the task graph into joint control instructions of the target robot arm. The control instruction sending unit is communicatively connected to the control instruction conversion unit and configured to send the joint control instructions to the joint controller for execution.

[0019] In a third aspect, the present application provides a control device comprising a storage module, a processing module and a transceiver module which are sequentially communicatively connected, wherein the storage module is configured to store a computer program, the transceiver module is configured to transceive messages, and the processing module is configured to read the computer program and execute the natural language driven robot arm control method as described in the first aspect or any possible design of the first aspect.

[0020] In a fourth aspect, the present application provides a computer readable storage medium having instructions stored thereon, wherein when the instructions are run on a computer, the natural language driven robot arm control method as described in the first aspect or any possible design of the first aspect is executed.

[0021] In a fifth aspect, the present application provides a computer program product comprising computer programs or instructions which, when executed by a computer, implement the natural language driven robot arm control method as described in the first aspect or any possible design of the first aspect.

[0022] The above-mentioned scheme has the following beneficial effects: (1) The present application creatively provides a new scheme for natural language driven robot arm control combining large language model, multi-modal perception and adaptive control technology, that is, after obtaining natural language instructions and multi-modal perception data, the natural language instructions are first decomposed into semantic embedding vectors, and a large language model is used for hierarchical semantic parsing to obtain structured task representation, then multi-modal perception data are fused to generate environment representation required for task execution, then the structured task representation and the environment representation are converted into a task graph, and the task graph is converted into joint control instructions of the target robot arm, finally the joint control instructions are sent to the joint controller for execution, thus through multi-level semantic understanding, multi-modal perception fusion and adaptive execution control, the robot arm can accurately understand and efficiently execute complex natural language instructions, and the task execution precision, adaptability, flexibility, ease of use and intelligent level in dynamic environment are improved; (2) It has high precision characteristics: the control error of the robot arm end effector can be less than 0.1mm, which is greatly improved compared with the 1mm error of the traditional method, and meets the high precision task demand; (3) It has strong adaptability: through the multi-modal perception fusion network, visual and tactile multi-source data are integrated in real time, and combined with the dynamic task graph generation algorithm, it can adapt to dynamic environment, support complex scenes such as light change, object occlusion and dynamic obstacles, etc. (4) It has intelligent characteristics: through hierarchical semantic parsing, the understanding accuracy of natural language instructions can be more than 95%, and it can process multi-step, fuzzy or context-related instructions, and improve the naturalness and efficiency of human-computer interaction (5) It has high efficiency: through real-time feedback adjustment of task execution, adaptive learning and improvement of robot arm control success rate are realized, specifically, the task completion time can be shortened by about 20%, and the error rate can be reduced to below 5%, thereby effectively improving the robustness; (6) It has expandability: the present application scheme can be designed in a modular way, and supports different types of robot arms and sensor configurations, which is convenient for extension to new tasks and scenes; (7) Can be suitable for various scenarios that require natural language driving and dynamic task execution, and is widely used in intelligent manufacturing, service robots, and medical assistance, etc. For example, in the field of intelligent manufacturing, the mechanical arm can accurately grasp the parts and complete the assembly according to the natural language instruction, effectively improving the flexibility and automation level of the production line. The mechanical arm can also perform quality detection tasks for product surface defects, and realize high-precision and high-efficiency detection through instruction driving. In the field of service robots, the mechanical arm can respond to the user's language instruction to complete household tasks such as fetching, arranging or cleaning, greatly improving the convenience of home life. In public service places such as hotels or hospitals, the mechanical arm can undertake the work of delivering goods or nursing assistance, helping to reduce the burden of manpower. In the field of medical assistance, the mechanical arm can accurately position and operate surgical instruments under the instruction of the doctor, improving the precision and safety of minimally invasive surgery. At the same time, it can also be used to provide personalized grip training or life support for rehabilitation patients, helping the rehabilitation process. (8) Can improve the accuracy and adaptability of task execution while enhancing the processing capability of dynamic environment. Experiments and application examples show that this scheme has certain practicability in intelligent manufacturing, service robots and medical assistance, etc. It can provide a new implementation path for natural language interaction control in related fields, and is convenient for practical application and promotion. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0024] Figure 1 The flowchart of the natural language driven mechanical arm control method provided by the embodiment of the present application.

[0025] Figure 2 The communication connection relationship diagram of the control device, the language instruction input device, the multi-modal perception device and the joint controller of the target mechanical arm provided by the embodiment of the present application.

[0026] Figure 3 The process example diagram of the hierarchical semantic analysis processing provided by the embodiment of the present application.

[0027] Figure 4 The structure diagram of the natural language driven mechanical arm control device provided by the embodiment of the present application.

[0028] Figure 5 The structure diagram of the control device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these embodiments without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0030] It should be understood that although the terms "first" and "second", etc., may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another. For example, the first object may be referred to as the second object, and similarly, the second object may be referred to as the first object, without departing from the scope of the exemplary embodiments of the invention.

[0031] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, or A and B exist simultaneously. Another example is A, B and / or C, which can mean that any one of A, B, and C or any combination thereof exists. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone or A and B exist simultaneously. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0032] Example like Figures 1 to 3 As shown, the natural language-driven robotic arm control method provided in the first aspect of this embodiment can be executed, but is not limited to, by a control device with certain computing resources. The control device is communicatively connected to a language command input device, a multimodal sensing device, and a joint controller of the target robotic arm. The multimodal sensing device is located in the surrounding environment of the target robotic arm, such as... Figure 2The language instruction input device is configured to input natural language instructions of a user, which can be implemented by a microphone combined with automatic speech recognition (ASR) technology, but is not limited thereto. The multi-modal perception device is configured to collect multi-modal perception data in real time, including but not limited to visual data, tactile data, and environmental data, etc., and specifically includes but is not limited to a visual sensor (such as an RGB-D camera) for collecting the visual data, a tactile sensor (such as a force sensor) for collecting the tactile data, and an environmental sensor (such as an inertial measurement unit, IMU sensor) for collecting the environmental data, etc. The target robot arm is a control object, wherein the joint controller is configured to generate a corresponding control signal according to the joint control instruction from the control device, and transmit the control signal to an actuator of the target robot arm for execution; the target robot arm and the joint controller can be implemented by using existing configurations.

[0033] As shown in Figure 1 The natural language driven robot arm control method includes but is not limited to the following steps S1-S6.

[0034] S1. Obtain natural language instructions input by the language instruction input device and multi-modal perception data collected by the multi-modal perception device, wherein the multi-modal perception data includes but is not limited to visual data, tactile data, and environmental data, etc.

[0035] In the step S1, since the subsequent joint control instruction is to be generated based on the natural language instructions and the multi-modal perception data, in order to ensure the rationality of the joint control instruction, the input time of the natural language instructions and the collection time of the multi-modal perception data need to be contemporaneous (i.e. within the same unit period), and preferably at the same time.

[0036] S2. Decompose the natural language instructions into semantic embedding vectors, and perform hierarchical semantic parsing processing on the semantic embedding vectors by using a large language model based on a Transformer structure to obtain a structured task representation, wherein the hierarchical semantic parsing processing includes but is not limited to sequentially performing intent recognition layer semantic parsing processing, parameter extraction layer semantic parsing processing, and task combination layer semantic parsing processing, etc., and the structured task representation includes but is not limited to a task intent obtained by the intent recognition layer semantic parsing processing, a task parameter obtained by the parameter extraction layer semantic parsing processing, and at least one identified sub-task and an execution order of the at least one identified sub-task obtained by the task combination layer semantic parsing processing, etc.

[0037] In the step S2, as Figure 3As shown, the process of decomposing the natural language instruction into a semantic embedding vector specifically but not limited to includes: first performing a regular text segmentation processing on the natural language instruction, and then performing a regular semantic embedding processing on the segmentation result to obtain the semantic embedding vector. The Transformer structure is an existing neural network architecture based on a self-attention mechanism, the core structure of which includes an encoder (Encoder) and a decoder (Decoder) two parts, mainly used for processing sequence data (such as natural language); the large language model (Large Language Model, LLM for short) is a kind of deep learning model trained by massive text data, which can generate natural language text or understand language meaning; thus the large language model can but not limited to adopt the existing BERT (Bidirectional Encoder Representations from Transformers, Bidirectional Encoder Representations from Transformers) model or GPT (Generative Pre-trained Transformer, Generative Pre-trained Transformer) model, etc., and combined with the field-specific prompt engineering and few-shot learning technology (that is, a machine learning technology, which aims to make the model quickly adapt to new tasks or identify new categories through a small amount of labeled data, such as 50 or less) to perform the hierarchical semantic parsing processing on the semantic embedding vector: first identify the overall intention of the instruction (such as "grabbing intention", "placing intention" or "moving intention", etc.), and then refine to specific parameters layer by layer (such as the grabbing target object of "grabbing intention", the placing target position of "placing intention" or the moving constraint condition of "moving intention", etc.), and also support to identify at least one of the identified sub-tasks (that is, the identified sub-tasks) in the multi-step instruction (such as "put the cup on the right side of the table") and their execution order, such as Figure 3 As shown.

[0038] In the step S2, specifically, in the process of hierarchical semantic parsing, the loss function of semantic parsing is represented as follows:

[0039] In the formula, represents the total number of natural language instruction samples, represents a positive integer less than or equal to , and represents the real semantic label (that is, the real required structured task representation) corresponding to the th natural language instruction sample, represents the th natural language instruction sample, model parameters of the large language model, a probability distribution function predicted by the large language model. In addition, as Figure 3 indicated, while obtaining the structured task representation, corresponding confidence evaluation information, such as task feasibility and parameter integrity, etc., can also be outputted, so as to trace back the history.

[0040] S3. Fusing the multi-modal perception data to generate an environment representation required for task execution.

[0041] In the step S3, in order to enhance the adaptability of the environment representation to task requirements and environmental changes, preferably, the multi-modal perception data is fused, including but not limited to: using a multi-modal dynamic fusion network based on a self-attention mechanism to optimize the modal contribution of each modal perception data in the multi-modal perception data according to task requirements and environmental changes in real time, and then fusing the multi-modal perception data according to all the modal contributions, wherein the modal contribution adopts a fusion weight represented by the following formula:

[0042] In the formula, denotes the total number of modes of the multi-modal perception data, and respectively denote positive integers less than or equal to , denotes the fusion weight of the th modal perception data in the multi-modal perception data, denotes a natural exponential function, denotes a preset adjustment parameter, denotes the relevance of the th modal perception data to a target task and has , denotes the feature vector of the th modal perception data, denotes a transpose symbol, denotes the feature vector of the target task, that is, the structured task representation, denotes the confidence of the th modal perception data, denotes a Sigmoid function, denotes the sensitivity of the th modal perception data to environmental changes and has , denotes a preset environmental change sensitivity coefficient, denotes an environment feature vector at a current time, denotes an environment feature vector at a previous time, In the multimodal sensing data, the first... The correlation between each modal sensing data and the target task. Indicates the first The sensitivity of each modal sensing data point to environmental changes. Specifically, the adjustment parameters... The recommended range is [2.0, 10.0], and the environmental change sensitivity coefficient is... The recommended range is [0.5, 2.0], and the feature vector... and the confidence level The data can be obtained from the output of a sensing device of the corresponding modality. The current time refers to the current acquisition time of the multimodal sensing data, and the previous time refers to the previous acquisition time of the multimodal sensing data. The environmental feature vector can be obtained by performing conventional feature extraction processing on the multimodal sensing data. In addition, the Sigmoid function is an existing function, commonly used to normalize confidence scores.

[0043] S4. The structured task representation and the environment representation are transformed into a task graph, wherein the nodes in the task graph represent the identified subtasks, and the edges in the task graph represent the execution order.

[0044] In step S4, the task graph is preferably a directed graph. Specifically, converting the structured task representation and the environment representation into a task graph includes, but is not limited to, using a task graph generation model based on a graph neural network to convert the structured task representation and the environment representation into a task graph, wherein the nodes in the task graph represent the identified subtasks, the edges in the task graph represent the execution order, and the task graph generation model needs to minimize the optimization objective. It is expressed as follows:

[0045] In the formula, This represents the total number of natural language instruction samples. Indicates less than or equal to positive integers, Indicates the relationship with the first Structured task representations corresponding to each natural language instruction sample. Indicates the relationship with the first The environment representation corresponding to the nth multimodal sensing data sample, the nth The first natural language instruction sample and the first It was obtained by acquiring multiple multimodal sensing data samples together. Indicates the first The first natural language instruction sample and the first The real task graph labels corresponding to each multimodal sensing data sample. This represents the probability distribution function predicted by the task graph generation model. This represents the preset complexity penalty coefficient. Indicates less than or equal to positive integers, This represents the number of nodes in the generated task graph. This represents the number of edges in the generated task graph, where the generated task graph refers to the graph generated by the task graph generation model that connects the first edge to the second edge. The structured task representation corresponding to the first natural language instruction sample and its relation to the first... The task graph is obtained by transforming the environment representation corresponding to each multimodal sensing data sample. The Graph Neural Network (GNN) is an algorithmic framework based on deep learning for processing graph-structured data. It performs tasks such as classification and prediction by extracting node, edge, and overall graph features, and is widely used in fields such as social networks and molecular structures. Therefore, the optimization objective can be applied in conventional applications. After optimizing the task graph generation model, the structured task representation and the environment representation can be transformed into a task graph, which can then dynamically adjust the task graph and support real-time environment changes.

[0046] S5. Convert the task diagram into joint control commands for the target robotic arm.

[0047] In step S5, to adaptively adjust constraints to ensure control accuracy and safety, preferably, the task diagram is converted into joint control commands for the target robotic arm, including but not limited to: during the conversion of the task diagram into joint control commands for the target robotic arm, an inverse kinematics algorithm capable of adaptively adjusting multiple constraints to solve the following optimization problem is used to obtain the joint angle vector used to generate the joint control commands:

[0048] In the formula, Indicates minimization. This represents the joint angle vector. Represents the target space coordinate vector. Represents the forward kinematic function. Represents positive integers. Represents the first in the multiple constraints The constraints are in functional form, and the multiple constraints include, but are not limited to, joint limit constraints, load constraints, and / or obstacle constraints. represents a preset constraint weight coefficient. The aforementioned forward kinematics (FK) is a basic problem in robotics, which is used to calculate the pose (position and attitude) of the end effector when the joint variables are known. The aforementioned inverse kinematics is a process of solving the joint parameters by knowing the target position and attitude of the movable object end, so the specific solving process can be derived based on the conventional technology, and the end positioning purpose is achieved.

[0049] S6. Sending the joint control instruction to the joint controller for execution.

[0050] The natural language driven robot arm control method based on the aforementioned steps S1-S6 provides a new scheme for natural language driven robot arm control combining large language model, multi-modal perception and adaptive control technology, that is, after obtaining the natural language instruction and multi-modal perception data, the natural language instruction is first decomposed into a semantic embedding vector, and a large language model is used for hierarchical semantic parsing to obtain a structured task representation, then the multi-modal perception data are fused to generate an environment representation required for task execution, then the structured task representation and the environment representation are converted into a task graph, and the task graph is converted into a joint control instruction of the target robot arm, finally the joint control instruction is sent to the joint controller for execution. Through multi-level semantic understanding, multi-modal perception fusion and adaptive execution control, the robot arm can accurately understand and efficiently execute complex natural language instructions, improve the task execution accuracy, adaptability, flexibility, ease of use and intelligent level in dynamic environment, and is particularly suitable for intelligent manufacturing, service robots and medical assistance, etc. Application scenarios that require high-precision human-computer interaction are convenient for practical application and promotion.

[0051] The embodiment provides a possible design for closed-loop feedback optimization based on the technical solution of the first aspect, that is, after the joint control instruction is sent to the joint controller for execution, the method further includes but is not limited to steps S7-S8.

[0052] S7. Collecting the execution state monitoring result of the target robot arm.

[0053] In the step S7, the execution state monitoring result can be obtained by a conventional collection of related sensors and / or a conventional feedback of a user.

[0054] S8. Using a reinforcement learning based controller to optimize the execution of the control task of the target robot arm according to the execution state monitoring result, wherein the reward function of the reinforcement learning represents as follows:

[0055] In the formula, 、 and respectively represent preset weight coefficients, represents the base of the natural logarithm, represents the control task accuracy of the target robot arm, represents the control task execution speed of the target robot arm, represents the number of execution errors of the control task of the target robot arm.

[0056] In the step S8, the reinforcement learning (RL) is a machine learning method, and its basic framework is a Markov decision process, which allows the agent to learn the optimal strategy through trial and error in the interaction with the environment. The control task accuracy (which can be in millimeters), the control task execution speed (which can be in seconds), and the number of execution errors can be regularly extracted according to the execution state monitoring result. Thus, based on the reinforcement learning and the reward function , the optimized execution of the control task of the target robot arm can be realized according to the execution state monitoring result, thereby achieving the purpose of adaptive learning and improving the success rate of robot arm control.

[0057] Thus, based on the foregoing possible design one, the task execution can also be adjusted in real time to achieve the purpose of adaptive learning and improving the success rate of robot arm control. Specifically, the task completion time can be shortened by about 20%, and the error rate can be reduced to below 5%, thereby effectively improving the robustness.

[0058] As shown in Figure 4 , the second aspect of the present embodiment provides a virtual device for implementing the natural language driven robot arm control method of the first aspect or possible design one, which is suitable for being arranged in a control device, wherein the control device is respectively communicatively connected with a language instruction input device, a multi-modal perception device, and a joint controller of a target robot arm, and the multi-modal perception device is located in the surrounding environment of the target robot arm. The virtual device comprises an instruction data acquisition unit, a semantic analysis processing unit, a perception data fusion unit, a task graph conversion unit, a control instruction conversion unit, and a control instruction sending unit. The instruction data acquisition unit is configured to acquire natural language instructions input by the language instruction input device and multi-modal perception data collected by the multi-modal perception device, wherein the multi-modal perception data comprises visual data, tactile data, and environmental data. The semantic analysis processing unit is in communication connection with the instruction data acquisition unit, configured to decompose the natural language instruction into a semantic embedding vector, and perform hierarchical semantic analysis processing on the semantic embedding vector by using a large language model based on a Transformer structure to obtain a structured task representation, wherein the hierarchical semantic analysis processing includes sequentially performed intent recognition layer semantic analysis processing, parameter extraction layer semantic analysis processing, and task combination layer semantic analysis processing, and the structured task representation includes a task intent obtained through the intent recognition layer semantic analysis processing, a task parameter obtained through the parameter extraction layer semantic analysis processing, and at least one identified subtask and an execution order of the at least one identified subtask obtained through the task combination layer semantic analysis processing. The perception data fusion unit is in communication connection with the instruction data acquisition unit, configured to fuse the multi-modal perception data to generate an environment representation required for task execution. The task graph transformation unit is in communication connection with the semantic analysis processing unit and the perception data fusion unit respectively, configured to transform the structured task representation and the environment representation into a task graph, wherein a node in the task graph represents the identified subtask, and an edge in the task graph represents the execution order. The control instruction transformation unit is in communication connection with the task graph transformation unit, configured to transform the task graph into joint control instructions of the target robot arm. The control instruction sending unit is in communication connection with the control instruction transformation unit, configured to send the joint control instructions to the joint controller for execution.

[0059] In one possible design, a monitoring result collection unit and a reinforcement learning control unit are further included and in communication connection; The monitoring result collection unit is configured to collect an execution state monitoring result of the target robot arm after the joint control instructions are sent to the joint controller for execution. The reinforcement learning control unit is configured to use a reinforcement learning-based controller to optimize execution of a control task of the target robot arm according to the execution state monitoring result, wherein a reward function of the reinforcement learning is represented as follows:

[0060] In the formula, 、 and respectively represent preset weight coefficients, represents a base of a natural logarithm, represents a control task accuracy of the target robot arm, representing a control task execution speed of the target robot arm, representing a control task execution error number of the target robot arm.

[0061] The working process, working details and technical effects of the aforementioned device provided by the second aspect of the embodiment can be referred to the natural language driven robot arm control method of the first aspect or the possible design one, which will not be repeated here.

[0062] As Figure 5 The third aspect of the embodiment provides a control device for executing the natural language driven robot arm control method of the first aspect or the possible design one, as shown in the figure. The control device comprises a storage module, a processing module and a transceiver module which are sequentially connected in communication. The storage module is used for storing a computer program. The transceiver module is used for transceiving messages. The processing module is used for reading the computer program and executing the natural language driven robot arm control method of the first aspect or the possible design one. Specifically, the storage module can include, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a flash memory, a first input first output (FIFO) memory and / or a first input last output (FILO) memory, etc. The processing module can use, but is not limited to, a microprocessor with the model of STM32F105 series. In addition, the control device can further include, but is not limited to, a power module, a display screen and other necessary components.

[0063] The working process, working details and technical effects of the aforementioned control device provided by the third aspect of the embodiment can be referred to the natural language driven robot arm control method of the first aspect or the possible design one, which will not be repeated here.

[0064] The fourth aspect of the embodiment provides a computer readable storage medium storing instructions of the natural language driven robot arm control method of the first aspect or the possible design one, i.e. the computer readable storage medium stores instructions. When the instructions run on a computer, the natural language driven robot arm control method of the first aspect or the possible design one is executed. The computer readable storage medium refers to a carrier for storing data, which can include, but is not limited to, floppy disks, optical disks, hard disks, flash memories, USB flash drives and / or memory sticks, etc. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices.

[0065] The working process, working details and technical effects of the aforementioned computer readable storage medium provided by the fourth aspect of the embodiment can be referred to the natural language driven robot control method as described in the first aspect or the possible design one, and will not be described here.

[0066] The fifth aspect of the embodiment provides a computer program product, including a computer program or instructions, which, when executed by a computer, implements the natural language driven robot control method as described in the first aspect or the possible design one. Wherein, the computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices.

[0067] Finally, it should be noted that: the above only for the preferred embodiments of the present application, and not for limiting the scope of protection of the present application. Any modification, equivalent replacement, improvement and decoration, etc. made within the spirit and principles of the present application, shall be included in the scope of protection of the present application.

Claims

1. A natural language driven robot arm control method, characterized by, is executed by a control device, wherein the control device is communicatively connected with a language instruction input device, a multi-modal perception device and a joint controller of a target robot arm respectively, and the multi-modal perception device is located in a surrounding environment of the target robot arm; The natural language driven robot arm control method comprises: acquiring natural language instructions input by the language instruction input device and multi-modal perception data collected by the multi-modal perception device, wherein the multi-modal perception data comprises visual data, tactile data and environmental data; decomposing the natural language instructions into semantic embedding vectors, and performing hierarchical semantic parsing processing on the semantic embedding vectors by using a large language model based on a Transformer structure to obtain a structured task representation, wherein the hierarchical semantic parsing processing comprises sequentially performing intent recognition layer semantic parsing processing, parameter extraction layer semantic parsing processing and task combination layer semantic parsing processing, and the structured task representation comprises a task intent obtained by the intent recognition layer semantic parsing processing, a task parameter obtained by the parameter extraction layer semantic parsing processing, at least one identified subtask obtained by the task combination layer semantic parsing processing and an execution order of the at least one identified subtask; fusing the multi-modal perception data to generate an environment representation required for task execution; transforming the structured task representation and the environment representation into a task graph, wherein nodes in the task graph represent the identified subtasks, and edges in the task graph represent the execution order; transforming the task graph into joint control instructions of the target robot arm; sending the joint control instructions to the joint controller for execution.

2. The natural language driven robot control method of claim 1, wherein, In the process of the layered semantic parsing processing, the loss function of semantic parsing is represented as follows: In the formula, denotes the total number of natural language instruction samples, denotes a positive integer less than or equal to , denotes the true semantic label corresponding to the th natural language instruction sample, denotes the th natural language instruction sample, denotes the model parameters of the large language model, denotes the probability distribution function predicted by the large language model.

3. The natural language driven robot control method of claim 1, wherein, The fusion of the multi-modal perception data comprises: using a multi-modal dynamic fusion network based on a self-attention mechanism to optimize the modal contribution of each modal perception data in the multi-modal perception data according to task requirements and environmental changes in real time, and then fusing the multi-modal perception data according to all the modal contributions, wherein the modal contribution adopts a fusion weight represented by the following formula: In the formula, denotes the total number of modalities of the multi-modal perception data, and respectively denote positive integers less than or equal to , denotes the fusion weight of the th modality perception data in the multi-modal perception data, denotes the natural exponential function, denotes a preset adjustment parameter, denotes the relevance of the th modality perception data to the target task and has , denotes the feature vector of the th modality perception data, denotes the transpose symbol, denotes the feature vector of the target task, that is, the structured task representation, denotes the confidence of the th modality perception data, denotes the Sigmoid function, denotes the sensitivity of the th modality perception data to environmental changes and has , denotes a preset environmental change sensitivity coefficient, denotes the environmental feature vector at the current moment, denotes the environmental feature vector at the previous moment, denotes the relevance of the th modality perception data in the multi-modal perception data to the target task, denotes the sensitivity of the th modality perception data to environmental changes.

4. The natural language driven robot control method of claim 1, wherein, The transformation of the structured task representation and the environment representation into a task graph comprises: a task graph generation model based on a graph neural network is employed to transform the structured task representation and the environment representation into a task graph, wherein nodes in the task graph represent the identified subtasks, edges in the task graph represent the execution order, the task graph generation model and an optimization objective to be minimized is represented as follows: In the formula, denotes the total number of natural language instruction samples, denotes a positive integer less than or equal to denotes the structured task representation corresponding to the th natural language instruction sample, denotes the environment representation corresponding to the th multimodal perception data sample, the th natural language instruction sample being obtained together with the th multimodal perception data sample, denotes the true task graph label corresponding to the th natural language instruction sample and the th multimodal perception data sample, denotes the probability distribution function predicted by the task graph generation model, denotes a preset complexity penalty coefficient, denotes a positive integer less than or equal to denotes the number of nodes in the generated task graph, denotes the number of edges in the generated task graph, the generated task graph being a task graph obtained by converting the structured task representation corresponding to the th natural language instruction sample and the environment representation corresponding to the th multimodal perception data sample by using the task graph generation model.​​ 5. The natural language driven robot control method of claim 1, wherein, The transformation of the task graph into joint control instructions of the target robot arm comprises: In the process of transforming the task graph into the joint control instructions of the target robot arm, an inverse kinematics algorithm capable of multi-constraint adaptive adjustment is used to solve the joint angle vector for generating the joint control instructions to solve the following optimization problem: In the formula, Indicates minimization. This represents the joint angle vector. Represents the target space coordinate vector. Represents the forward kinematic function. Represents positive integers. Represents the first in the multiple constraints The constraints are in functional form, and the multiple constraints include joint limit constraints, load constraints, and / or obstacle constraints. This represents the preset constraint weight coefficient.

6. The natural language driven robot control method of claim 1, wherein, After the joint control instructions are sent to the joint controller for execution, the method further comprises: collecting execution state monitoring results of the target robot arm; a controller based on reinforcement learning is adopted to optimize the execution of the control task of the target robot arm according to the execution state monitoring result, wherein a reward function of the reinforcement learning is represented as follows: ​ In the formula, 、 and respectively represent preset weight coefficients, represents the base of the natural logarithm, represents the control task accuracy of the target mechanical arm, represents the control task execution speed of the target mechanical arm, represents the number of execution errors of the control task of the target mechanical arm.

7. A natural language driven robot arm control apparatus, characterized by, is applicable to be arranged in a control device, wherein the control device is communicatively connected with a language instruction input device, a multi-modal perception device and a joint controller of a target robot arm respectively, and the multi-modal perception device is located in a surrounding environment of the target robot arm; The natural language driven robot arm control device comprises an instruction data acquisition unit, a semantic analysis processing unit, a perception data fusion unit, a task graph conversion unit, a control instruction conversion unit and a control instruction sending unit; The instruction data acquisition unit is configured to acquire natural language instructions input by the language instruction input device and multi-modal perception data collected by the multi-modal perception device, wherein the multi-modal perception data comprises visual data, tactile data and environmental data; The semantic analysis processing unit is communicatively connected to the instruction data acquisition unit and configured to decompose the natural language instructions into semantic embedding vectors and perform hierarchical semantic analysis processing on the semantic embedding vectors by using a large language model based on a Transformer structure to obtain a structured task representation, wherein the hierarchical semantic analysis processing comprises sequentially performing intent recognition layer semantic analysis processing, parameter extraction layer semantic analysis processing and task combination layer semantic analysis processing, and the structured task representation comprises a task intent obtained by the intent recognition layer semantic analysis processing, a task parameter obtained by the parameter extraction layer semantic analysis processing, at least one identified subtask obtained by the task combination layer semantic analysis processing and an execution order of the at least one identified subtask; The perception data fusion unit is communicatively connected to the instruction data acquisition unit and configured to fuse the multi-modal perception data to generate an environmental representation required for task execution; The task graph conversion unit is communicatively connected to the semantic analysis processing unit and the perception data fusion unit, respectively, and configured to convert the structured task representation and the environmental representation into a task graph, wherein nodes in the task graph represent the identified subtasks, and edges in the task graph represent the execution order; The control instruction conversion unit is communicatively connected to the task graph conversion unit and configured to convert the task graph into joint control instructions of the target robot arm; The control instruction sending unit is communicatively connected to the control instruction conversion unit and configured to send the joint control instructions to the joint controller for execution.

8. A control device characterized by comprising: The natural language driven robot arm control method comprises the following steps: acquiring natural language instructions input by a language instruction input device and multi-modal perception data collected by a multi-modal perception device, decomposing the natural language instructions into semantic embedding vectors, performing hierarchical semantic analysis processing on the semantic embedding vectors by using a large language model based on a Transformer structure to obtain a structured task representation, fusing the multi-modal perception data to generate an environmental representation required for task execution, converting the structured task representation and the environmental representation into a task graph, converting the task graph into joint control instructions of a target robot arm, and sending the joint control instructions to a joint controller for execution.

9. A computer-readable storage medium, characterized in that The computer readable storage medium stores instructions, and when the instructions are executed on a computer, the natural language driven robot arm control method is executed.

10. A computer program product comprising computer programs or instructions, characterized in that, The computer program or the instructions realize the natural language driven robot arm control method when executed on a computer.

Images