Robot programming method and device, storage medium, demonstrator and program product

Abstract

The invention provides a robot programming method and device, a storage medium, a demonstrator and a program product. The method comprises the steps of receiving a task description statement; analyzing the received task description statement through a pre-trained natural language processing model, and extracting an action keyword, an action parameter and a sequence identification word; constructing a semantic analysis tree according to the extracted action keywords, action parameters and sequence identification words, and determining an action sequence and a logic dependency relationship; converting the task description statement into a robot instruction according to the extracted action keyword and action parameter and the determined action sequence and logic dependency relationship; and generating a robot program code which can be executed by the target robot according to the converted robot instruction. According to the scheme provided by the invention, the robot programming threshold can be reduced, a user does not need to master a professional programming language, the programming efficiency is improved, and manual one-by-one instruction input is avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the control field, and particularly to a programming method and device of a robot, a storage medium, a teach pendant and a program product. BACKGROUND

[0002] In the related art, industrial robot programming relies on manual input of instructions by a teach pendant (such as point teaching, logic sentence writing), and requires the operator to have professional programming skills, which is low in efficiency and prone to errors. In the related art, the teach pendant only provides a graphical interface or script language support, and cannot directly generate a program through natural language description of task requirements. SUMMARY

[0003] The main purpose of the present application is to overcome the defects of the above-mentioned related art, and to provide a programming method and device of a robot, a storage medium, a teach pendant and a program product, so as to solve the problem that in the related art, industrial robot programming relies on manual input of instructions by a teach pendant, and requires the operator to have professional programming skills, which is low in efficiency and prone to errors.

[0004] In one aspect, the present application provides a programming method of a robot, comprising: receiving a task description sentence; analyzing the received task description sentence through a pre-trained natural language processing model, and extracting action keywords, action parameters and sequence identifier words; constructing a semantic analysis tree according to the extracted action keywords, action parameters and sequence identifier words, and determining the action sequence and logical dependency relationship; converting the task description sentence into a robot instruction according to the extracted action keywords and action parameters and the determined action sequence and logical dependency relationship; and generating a robot program code executable by a target robot from the converted robot instruction.

[0005] Optionally, it further comprises: before generating the executable robot program code of the target robot from the converted robot instruction, adding necessary parameters not involved in the task description sentence and / or performing parameter format conversion in the converted robot instruction according to a preset syntax rule.

[0006] Optionally, the necessary parameters comprise default parameters and / or context-derived parameters.

[0007] The parameter format conversion comprises converting a position description in the task description sentence into a robot coordinate.

[0008] Optionally, it further comprises: in the generation stage of the robot program code, detecting whether there is a conflict in the generated code in real time; and if a conflict is detected in the generated code, correcting it according to a preset logic.

[0009] Optionally, receiving a task description sentence comprises: providing a text box, and receiving a task description sentence in the text box.

[0010] Optionally, the action order and logical dependency relationship are determined by constructing a semantic parsing tree according to the extracted action keywords, action parameters and order identification words, including: creating a node for each identified action, and creating a sub-node or attribute according to the corresponding action parameters and / or order identification words, to construct a semantic parsing tree, so as to determine the order and logical dependency relationship.

[0011] In another aspect, the application provides a programming device for a robot, including: a receiving unit configured to receive a task description sentence; an analyzing unit configured to analyze the task description sentence received by the receiving unit by using a pre-trained natural language processing model, and extract action keywords, action parameters and order identification words; a determining unit configured to construct a semantic parsing tree according to the action keywords, action parameters and order identification words extracted by the analyzing unit, and determine an action order and a logical dependency relationship; a converting unit configured to convert the task description sentence into a robot instruction according to the action keywords and action parameters extracted by the analyzing unit and the action order and logical dependency relationship determined by the determining unit; and a generating unit configured to generate a robot program code executable by a target robot from the robot instruction converted by the converting unit.

[0012] Optionally, the device further includes an adding unit configured to add necessary parameters not involved in the task description sentence and / or perform parameter format conversion in the converted robot instruction according to a preset syntax rule before the converted robot instruction is generated into an executable robot program of the target robot.

[0013] Optionally, the necessary parameters include default parameters and / or context-derived parameters, and the parameter format conversion includes converting a position description in the task description sentence into a robot coordinate.

[0014] Optionally, the device further includes a detecting unit configured to detect whether a conflict exists in the generated code in real time during the generation of the robot program code, and a correcting unit configured to correct the generated code according to a preset logic if the detecting unit detects that a conflict exists in the generated code.

[0015] Optionally, the receiving unit receives the task description sentence by providing a text box in which the task description sentence is received.

[0016] Optionally, the determining unit constructs a semantic analysis tree according to the action keywords, action parameters and order identification words extracted by the parsing unit, and determines the action order and logical dependency relationship, including: creating a node for each identified action, and creating a sub-node or attribute according to the corresponding action parameters and / or order identification words, constructing a semantic analysis tree, and thus determining the order and logical dependency relationship.

[0017] In still another aspect, the present application provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the preceding methods.

[0018] In still another aspect, the present application provides a demonstrator comprising a processor, a memory, and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of any of the preceding methods.

[0019] In still another aspect, the present application provides a demonstrator comprising the programming device of any of the preceding robots.

[0020] In still another aspect, the present application provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of any of the preceding methods.

[0021] According to the technical solution of the present application, natural language description is supported, and natural language can be converted into logical steps (such as motion trajectory, IO control, and loop branching) executable by a robot, so as to reduce the programming threshold of the robot and avoid the need for the user to master a professional programming language.

[0022] According to the technical solution of the present application, a task description sentence can be automatically converted into executable robot instructions, the program generation efficiency in a complex scene is improved, the programming efficiency is improved, manual input of instructions is avoided, and the application programming cycle is shortened.

[0023] According to the technical solution of the present application, the NLP model (BERT variant injecting robot kinematics parameters) based on domain knowledge enhancement can accurately extract action intention, coordinate parameters and logical conditions from text, improve the program generation efficiency in a complex scene, improve the programming efficiency, avoid manual input of instructions, and reduce the additional hardware caused by multi-modal recognition and increase the system complexity problem.

[0024] According to the technical solution of the present application, a standardized program is automatically generated according to the syntax rules (such as KUKA KRL, ABB RAPID, and Gree robot) of a target robot, and the adaptability problem (such as ignoring brand syntax differences and kinematics constraints) of a general code generation model in the field of robots is solved.

[0025] According to the technical scheme of the present application, the embedded logic analysis engine is introduced, and the singular points and the space unreachable points are specially processed, so that the program error is avoided and repeatedly debugged, and the program error can be reduced through semantic analysis and logic verification. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0027] Figure 1 is a method schematic diagram of an embodiment of the programming method of the robot provided by the present application;

[0028] Figure 2 The language analysis process of the present application is shown;

[0029] Figure 3 The system flowchart of the programming method of the robot provided by the present application is shown

[0030] Figure 4 is a structure block diagram of an embodiment of the programming device of the robot provided by the present application. DETAILED DESCRIPTION

[0031] In order to make the objects, technical schemes and advantages of the present application clearer, the technical schemes of the present application will be described below in conjunction with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0033] The present application provides a programming method of a robot.

[0034] Figure 1It is a method schematic diagram of an embodiment of the robot programming method provided by the present invention.

[0035] like Figure 1 As shown, according to one embodiment of the present invention, the robot programming method includes at least step S110, step S120, step S130, step S140 and step S150.

[0036] Step S110: receiving a task description statement.

[0037] Specifically, a text box is provided in which a task description statement is received. That is, a text box can be provided for the user to enter a task description. For example, the user enters "the robotic arm moves from the Home point to (X=100, Y=200) at 50% speed, grabs an object, and places it on the conveyor belt."

[0038] Step S120 , parsing the received task description statement through a pre-trained natural language processing model to extract action keywords, action parameters and sequence identifiers.

[0039] Specifically, a pre-trained NLP model (natural language processing model) is injected with a BERT variant of the robot's kinematic parameters to parse the task description sentence and extract action keywords and action parameters. More specifically, the pre-trained NLP model is used to perform intent recognition and entity recognition on the received task description sentence, extracting action keywords (e.g., move, clamp), action parameters (e.g., coordinate parameters, speed values) and sequence identifiers (e.g., "first", "then", "last", "next", "after", etc.), that is, breaking the input task description sentence into action + parameters.

[0040] Intent recognition, that is, identifying the intent type of the task description sentence, such as move, grasp, place, etc. For example, BERT or RoBERTa models can be used for classification and output intent labels, such as MOVE_ACTION and GRASP_ACTION.

[0041] Entity recognition is to extract parameter entities in task description sentences, such as position coordinates (such as "reach x=30, y=50"), action objects (such as "grab the red building block"), speed (such as "move quickly"), and time conditions (such as "then" and "wait a while").

[0042] For example, the input sentence is: "First, pick up the red block, then move it to coordinates x=20, y=35, and finally put it down at a medium speed." The entity recognition result is:

[0043]

[0044] Step S130, constructing a semantic parsing tree according to the extracted action keywords, action parameters and order identification words, determining the action order and logical dependency relationship.

[0045] Specifically, a node is created for each recognized action, and a subnode or attribute is created according to the corresponding action parameter and / or order identification word, and a semantic parsing tree is constructed to determine the action order and logical dependency relationship.

[0046] First, according to the order identification words (such as "first", "then", "last", "next", "after" and the like) extracted by the natural language processing model (NLP model), the execution order between actions is determined.

[0047] For example:

[0048] "First X, then Y": construct a directed edge X->Y.

[0049] "Complete X, then perform Y": X->Y

[0050] Then, according to the determined execution order between actions, the dependency graph is built. That is, the actions corresponding to the recognized action keywords are constructed as nodes, and the dependency relationship is represented in the form of a directed graph.

[0051] For example, the above sentence "first pick up the red block, then move to coordinates x=20, y=35, and finally drop it at medium speed." generates a directed graph:

[0052] [pick up] -> [move] -> [drop]

[0053] The generated directed graph can be parsed in the robot executor in the form of graph structure or queue scheduling.

[0054] If a logical condition is involved (such as "if the pick-up fails, then..."), the following structure can be constructed:

[0055] [ACTION: pick up]

[0056] └─OBJECT: red block

[0057] └─ON_FAIL: [ACTION: retry pick up]

[0058] For example, for the task description sentence "first pick up the red block, then move to coordinates x=20, y=35, and finally drop it at medium speed.", the nodes of "pick up", "move" and "drop" can be constructed as follows:

[0059] [ACTION: pick up]

[0060] └─OBJECT: red block

[0061] [ACTION: move]

[0062] ├-POSITION: (20, 35)

[0063] └-ORDER: after grasp

[0064] [ACTION: drop]

[0065] ├-SPEED: medium

[0066] └-ORDER: after move

[0067] In this way, a preliminary semantic parsing tree structure is formed, each action being a node, with action parameters and order information.

[0068] Figure 2 A language parsing process of the application is shown. Figure 2 As shown, first, word segmentation is performed, then intent recognition and parameter extraction are performed, and then a semantic parsing tree is constructed.

[0069] The application uses an NLP model to perform extraction analysis according to input keywords, then uses robot instructions for correspondence, then extracts parameters, stores using variables, and then constructs a logic tree.

[0070] In step S140, the task description sentence is converted into robot instructions according to the extracted action keywords and action parameters and the determined action order and logical dependency relationship.

[0071] Specifically, different intents (actions) map different robot instructions.

[0072] The following are examples of intent mapping instructions:

[0073] GRASP SetDO (set output port) SetDO DO[1] = 1 (open gripper)

[0074] MOVE MovL (linear movement) MovL P[1]V = 100 PL = 0

[0075] WAIT WaitDI (wait for input) WaitDIDI[1] == 1

[0076] PLACE SetDO (close gripper) SetDO DO[1] = 0

[0077] Preferably, after converting the task description sentence into robot instructions, according to the preset grammar rules, necessary parameters not involved in the task description sentence are added in the converted robot instructions and / or parameter format conversion is performed.

[0078] Specifically, different models of robots (e.g. different robot brands such as ABB, UR, KUKA, DOBOT) have different requirements for instruction formats. Based on the action parameters identified by the natural language processing model NLP, necessary parameters not involved in the task description statement (i.e. not set by the user) are added and / or parameter format conversion is performed.

[0079] The necessary parameters may include, for example, default parameters and / or context-derived parameters. Default parameters may include, for example, speed and acceleration default values (e.g. V = 100, A = 50 when not mentioned); tool coordinate system default set to Tool = 0; work coordinate system default set to Base = 0. Context-derived parameters are mainly used to fill in parameters within instructions in the robot program, for example, it has been determined according to the context that the robot instruction needs to implement linear motion using MovL, and according to the user input context (if the user context does not mention it, the default parameters of the corresponding brand robot instruction are filled in), the filled-in parameters are MovL (P0, V100, Tool0, Wobj0).

[0080] The parameter format conversion may include, for example, converting the position description in the task description statement into robot coordinates.

[0081] For example, the position is mapped to a robot coordinate point (P point)

[0082] For example, if the input is x = 50, y = 30, it should be converted to:

[0083] P[1] = {X 50.0, Y 30.0, Z 10.0, Rx 0, Ry 0, Rz 0}

[0084] Subsequently, instructions can be generated according to the robot language format used:

[0085] Example - DOBOT Magician Lite syntax:

[0086] MovL P[1] V = 100 A = 50 Tool = 0

[0087] Example - UR Script syntax:

[0088] movej(p[0.050,0.030,0.010,0,0,0],a=1.2,v=0.25)

[0089] Step S150, the converted robot instructions are generated into a robot program executable by the target robot.

[0090] Specifically, a preset template framework is called to generate a standardized program, i.e., the template framework is called to generate an executable robot program of a target robot from the converted robot instruction. More specifically, the structured "action intention + parameters" are mapped into executable program code conforming to the syntax of the target robot using the template framework. The template framework, i.e., a template program of robot programming, is composed of parameter declaration, main program and instruction. The parameter declaration and instruction generate different contents according to the user input content, and the function declaration is fixed content.

[0091] For example, the data is fitted into a preset template framework,

[0092] For example, the data is fitted into a preset template framework,

[0093] MovL({{point}},V={{velocity}},A={{accel}},Tool={{tool}})

[0094] MovL({{point}},V={{velocity}},A={{accel}},Tool={{tool}})

[0095] MovL({{point}},V={{velocity}},A={{accel}},Tool={{tool}})

[0096] According to the user input description sentence, the parameters are added:

[0097] Parameter structure input (generated by semantic analysis and mapping stage)

[0098]

[0099] After the parameters are determined, the sentence is generated: MovL(P1, 50, 20, 0); SetDO(DO1, 1);

[0100] According to the above embodiment, the standardized program is automatically generated according to the syntax rules of the target robot (such as KUKA KRL, ABB RAPID, and Gre robot), solving the adaptability problem of the general code generation model in the field of robots (such as ignoring brand syntax differences and kinematic constraints).

[0101] Preferably, in the generation stage of the robot program code, it is detected in real time whether there is a conflict, and if there is a conflict, the conflict is corrected according to a preset logic.

[0102] The conflict can specifically include at least one of joint over-limit, undefined point, singular point and spatial unreachable point. If it is detected that there is a conflict, the conflict can be corrected according to a preset logic, for example, an intermediate point is automatically added for transition, so as to avoid the existence of the conflict in the generated code. The preset logic can be embedded logic verification engine generated by programming, so as to correct the conflict.

[0103] Take joint limit violations as an example: the target point position causes the joint angle to exceed the physical limit, and the target posture causes J4 to exceed +180°. A joint limit check is performed to determine whether a joint limit violation exists through inverse calculation and simulation verification. The target point (Cartesian coordinates) is input into the robot's inverse kinematic (IK) model to determine whether a legal solution exists and whether the joint angle constraints are satisfied. The code is as follows:

[0104]

[0105] For joint over-limit problems, you can insert intermediate points to split large movements; adjust the posture; and optimize the direction of the tool coordinate system (for example, modify the value of the tool coordinate system to change the coordinate system direction of the tool on the top of the robot).

[0106] According to the above embodiment, an embedded logic analysis engine is introduced to perform special processing on singular points and spatially inaccessible points, thereby avoiding repeated debugging due to program errors and reducing program errors.

[0107] The above steps of the present invention can also refer to Figure 3 As shown in the system flow chart. Figure 3 As described, the user inputs a task description statement, and the pre-trained NLP model performs intent classification and entity recognition, extracts action keywords, performs logical mapping, converts natural language instructions into basic robot operation units, performs embedded logic verification, detects conflicts in real time during the code generation stage, and calls the template engine to generate standardized programs.

[0108] In order to clearly illustrate the technical solution of the present invention, the execution process of the robot programming method provided by the present invention is described below with an example.

[0109] Take the use of robots in palletizing scenarios as an example:

[0110] 1. To implement the effect of robots transporting goods at different locations in a palletizing scenario, users can enter the following in the user input interface: "Palletizing items to be stacked in 3 rows, 3 columns, and 2 layers, with a starting point of (X:650.000; Y:100.000, Z:100.000; C:0.000), a row distance of 400, a column distance of 400, and a height of 200 to achieve the effect of stacking items transported by the conveyor belt."

[0111] 2. Combination Figure 2The semantic analysis process shown, first, the NLP model will be based on the task description information input in step 1 to carry out word segmentation, extract 3 rows, 3 columns, 2 layers, starting point information (X: 650.000; Y: 100.000, Z: 100.000; C: 0.000), row distance 400, column distance 400, height 200, in turn; Then, according to the stacking goods, the intention is recognized, and the SetPallet instruction is converted to form a semantic analysis tree: first, according to the origin information, row and column height value, configure the stacking, according to the stacking information, configure the stacking parameters, use the While loop instruction to run the stacking instruction of different points, and output the DO signal to inform the whole system that the stacking is completed.

[0112] 3. According to the logic of step 2, the conversion of the corresponding robot programming language is completed:

[0113] Table 1

[0114]

[0115] Table 1 shows the semantic analysis tree analysis result and the corresponding programming language. For example, when the user wants to realize the effect of the robot carrying goods at different points in the stacking scene, the user can input "stacking goods stack 3 rows 3 columns 2 layers, starting point (X: 650.000; Y: 100.000, Z: 100.000; C: 0.000), row distance 400, column distance 400, height 200" in the user input interface to realize the effect of stacking the goods transported by the conveyor belt. The step of generating the robot program is realized.

[0116] 4. If a singular point or a space unreachable point appears in the above process, the embedded logic verification engine is automatically added to the intermediate point for overdrive.

[0117] 5. Finally, the whole program is presented as code.

[0118] The application also provides a robot programming device.

[0119] Figure 4 is the structure block diagram of an embodiment of the robot programming device provided by the application. As Figure 4 shown, the robot programming device 100 includes a receiving unit 110, an analysis unit 120, a determination unit 130, a conversion unit 140 and a generation unit 150.

[0120] The receiving unit 110 is used for receiving the task description sentence.

[0121] Specifically, a text box is provided in which a task description sentence is received. That is, a text box can be provided for a user to input a task description, for example, the user inputs "the robot arm moves from the Home point to (X=100, Y=200) at 50% speed, and after picking up the object, places it on the conveyor belt."

[0122] The parsing unit 120 is configured to parse the task description sentence received by the receiving unit by using a pre-trained natural language processing model to extract action keywords, action parameters, and sequence identification words.

[0123] Specifically, a pre-trained NLP model (natural language processing model) is injected with a BERT variant of the robot kinematics parameters, which is used to parse the task description sentence and extract action keywords and action parameters. More specifically, the pre-trained NLP model is used for intent recognition and entity recognition on the received task description sentence to extract action keywords (such as move, grasp), action parameters (which can include coordinate parameters, speed values), and sequence identification words (such as "first", "then", "last", "next", "after", etc.), i.e., the input task description sentence is disassembled into action + parameter.

[0124] Intent recognition, i.e., identifying the intent type of the task description sentence, such as move, grasp, place, etc. For example, a BERT or RoBERTa model can be used for classification to output an intent label, such as MOVE_ACTION, GRASP_ACTION.

[0125] Entity recognition, i.e., extracting parameter entities in the task description sentence, such as position coordinates (e.g., "to x=30, y=50"), action objects (e.g., "grasp the red block"), speed (e.g., "move quickly"), and time conditions (e.g., "then", "wait a while").

[0126] For example, the input sentence is: "first grasp the red block, then move to coordinates x=20, y=35, and finally place it at medium speed." The entity recognition result is:

[0127]

[0128] The determining unit 130 is configured to construct a semantic parsing tree according to the action keywords, action parameters, and sequence identification words extracted by the parsing unit, and determine the action sequence and logical dependency relationship.

[0129] Specifically, a node is created for each recognized action, and a subnode or attribute is created according to the corresponding action parameters and / or sequence identification words to construct a semantic parsing tree, thereby determining the action sequence and logical dependency relationship.

[0130] First, the execution order between actions is determined according to the sequential identification words (e.g. "first", "then", "last", "next", "after" and the like) extracted by the natural language processing model (NLP model).

[0131] For example:

[0132] "First X, then Y": Build a directed edge of X->Y.

[0133] "After completing X, perform Y": X->Y

[0134] Then, the dependency graph is built according to the determined execution order between actions. That is, the actions corresponding to the recognized action keywords are constructed as nodes, and the dependency relationship is represented in the form of a directed graph.

[0135] For example, the above sentence "first pick up the red block, then move to coordinates x=20, y=35, and finally put it down at medium speed." generates a directed graph:

[0136] [pick up]→[move]→[put down]

[0137] The generated directed graph can be parsed in the robot executor in the form of graph structure or queue scheduling.

[0138] Among them, if it involves a logical condition (such as "if the pickup fails, then..."), the following structure can be constructed:

[0139] [ACTION: pick up]

[0140] └─OBJECT: red block

[0141] └─ON_FAIL: [ACTION: retry pick up]

[0142] For example: For the task description sentence "first pick up the red block, then move to coordinates x=20, y=35, and finally put it down at medium speed.", the nodes of "pick up", "move" and "put down" can be constructed as follows:

[0143] [ACTION: pick up]

[0144] └─OBJECT: red block

[0145] [ACTION: move]

[0146] ├─POSITION: (20, 35)

[0147] └─ORDER: after pick up

[0148] [ACTION: put down]

[0149] ├─SPEED: medium

[0150] └─ORDER: after movement

[0151] Thus, a preliminary semantic parsing tree structure is formed, each action being a node, with action parameters and dependency order information.

[0152] Figure 2 The language parsing process of the present application is shown. As shown, first, word segmentation is performed, then intent recognition and parameter extraction are performed, and then a semantic parsing tree is constructed. Figure 2

[0153] The present application uses an NLP model to perform extraction analysis according to input keywords, then uses robot instructions for correspondence, then extracts parameters, uses variable storage, and then constructs a logic tree.

[0154] The conversion unit 140 is configured to convert the task description sentence into a robot instruction according to the action keyword and action parameter extracted by the parsing unit and the action order and logical dependency relationship determined by the determination unit.

[0155] Specifically, different intents (actions) map different robot instructions.

[0156] The following is an example of intent mapping instructions:

[0157] GRASP SetDO (set output port) SetDO DO[1]=1 (open gripper)

[0158] MOVE MovL (linear movement) MovL P[1]V=100PL=0

[0159] WAIT WaitDI (wait for input) WaitDIDI[1]==1

[0160] PLACE SetDO (close gripper) SetDO DO[1]=0

[0161] Preferably, the device 100 further comprises an adding unit (not shown) for adding necessary parameters not involved in the task description sentence and / or performing parameter format conversion in the converted robot instruction according to a preset syntax rule before the converted robot instruction is generated into an executable robot program of a target robot by the conversion unit.

[0162] ​Specifically, different models of robots (e.g. robots of different robot brands, such as ABB, UR, KUKA, DOBOT) have different requirements for instruction formats. On the basis of the action parameters identified by the natural language processing model NLP, necessary parameters not involved in the task description statement (i.e. not set by the user) are added and / or parameter format conversion is performed.

[0163] The necessary parameters may include, for example, default parameters and / or context-derived parameters. The default parameters may include, for example, speed and acceleration default values (e.g. V = 100 and A = 50 when not mentioned); the tool coordinate system is set to Tool = 0 by default; and the work coordinate system is set to Base = 0 by default. The context-derived parameters are mainly used to fill in the parameters in the instructions in the robot program. For example, it has been determined according to the context that the robot instruction needs to implement linear motion using MovL, and the parameters are filled in as MovL (P0, V100, Tool0, Wobj0) according to the context input by the user (if the user context does not mention it, the default parameters of the corresponding brand robot instruction are filled in).

[0164] The parameter format conversion may include, for example, converting the position description in the task description statement into robot coordinates.

[0165] For example, the position is mapped to a robot coordinate point (P point)

[0166] For example, if the input is x = 50, y = 30, it should be converted to:

[0167] P[1] = {X 50.0, Y 30.0, Z 10.0, Rx 0, Ry 0, Rz 0}

[0168] Then, the instruction can be generated according to the robot language format used:

[0169] Example - DOBOT Magician Lite syntax:

[0170] MovL P[1] V = 100 A = 50 Tool = 0

[0171] Example - UR Script syntax:

[0172] movej(p[0.050,0.030,0.010,0,0,0],a = 1.2,v = 0.25)

[0173] The generation unit 150 is configured to generate robot program code executable by a target robot from the robot instruction converted by the conversion unit.

[0174] Specifically, the preset template framework is called to generate a standardized program, i.e., the template framework is called to generate an executable robot program of the target robot from the converted robot instruction. More specifically, the structured "action intention + parameter" is mapped into executable program code conforming to the syntax of the target robot using the template framework. The template framework, i.e., the template program of robot programming, is composed of parameter declaration, main program and instruction. The parameter declaration and instruction generate different contents according to the user input content, and the function declaration is fixed content.

[0175] For example, the data is fitted into the pre-set code template framework,

[0176] Taking MovL as an example (linear movement), the template framework is as follows:

[0177] MovL({{point}}, V = {{velocity}}, A = {{accel}}, Tool = {{tool}})

[0178] Clamp jaw operation SetDO:

[0179] SetDO DO[{{port}}] = {{value}}

[0180] According to the user input description sentence, the parameters are added:

[0181] Parameter structure input (generated by semantic analysis and mapping stage)

[0182]

[0183] After the parameters are determined, the sentence is generated: MovL(P1, 50, 20, 0); SetDO(DO1, 1);

[0184] Preferably, the device 100 further comprises a detection unit and a correction unit (not shown). The detection unit is configured to detect whether the generated code has conflicts in real time during the generation of the robot program code; and the correction unit is configured to correct according to a preset logic if the detection unit detects that the generated code has conflicts.

[0185] The conflicts can specifically include at least one of joint over-limit, undefined point, singular point and spatial unreachable point. If a conflict is detected, the preset logic can be used for correction, for example, an intermediate point is automatically added for transition, so as to avoid the conflicts in the generated code. The preset logic can be generated by writing a program to generate an embedded logic verification engine, so as to correct the conflicts.

[0186] Take joint limit as an example: the target point causes the joint angle to exceed the physical limit, and the target pose makes J4 exceed +180°. Joint limit check is performed through inverse solution calculation and simulation verification to determine whether there is joint limit. The target point (Cartesian coordinates) is input into the robot inverse solution (IK) model to determine whether there is a legal solution and whether the joint angle constraint is met. The code is as follows:

[0187]

[0188] For joint limit problems, intermediate points can be inserted to split large-scale actions, the pose can be adjusted, and the tool coordinate system direction can be optimized (for example, the value of the tool coordinate system is modified, and the coordinate system direction of the robot top tool is changed).

[0189] The application also provides a storage medium corresponding to the robot program generation method, which stores a computer program, and the program is executed by a processor to realize the steps of any of the above-mentioned methods.

[0190] The application also provides a teach pendant corresponding to the robot program generation method, which includes a processor, a memory, and a computer program stored on the memory and executable on the processor, and the processor executes the program to realize the steps of any of the above-mentioned methods.

[0191] The application also provides a teach pendant corresponding to the robot program generation method, which includes any of the above-mentioned robot program generation devices.

[0192] The application also provides a computer program product corresponding to the robot program generation method, which includes a computer program, and the computer program is executed by a processor to realize the steps of any of the above-mentioned methods.

[0193] Accordingly, the scheme provided by the application supports natural language description, can convert natural language into executable logic steps (such as motion trajectory, IO control, and loop branching) of the robot, and can reduce the programming threshold of the robot without the need for users to master professional programming languages.

[0194] According to the technical scheme of the application, the task description sentence can be automatically converted into executable robot instructions, and the application programming cycle is shortened.

[0195] According to the technical scheme of the application, the NLP model based on domain knowledge enhancement (BERT variant injecting robot kinematics parameters) can accurately extract action intent, coordinate parameters and logic conditions from text, improve the program generation efficiency in complex scenarios, improve the programming efficiency, avoid manual input of instructions, and reduce the additional hardware caused by multi-modal recognition and increase the system complexity problem.

[0196] According to the technical scheme of the application, a standardized program is automatically generated according to the syntax rules of a target robot (such as KUKA KRL, ABB RAPID, and GreRobot), and the adaptability problem of a general code generation model in the field of robots (such as ignoring brand syntax differences and kinematic constraints) is solved.

[0197] According to the technical scheme of the application, an embedded logic analysis engine is introduced, and special processing is performed on singular points and space unreachable points, error debugging of a program is avoided, and program errors can be reduced through semantic analysis and logic verification.

[0198] According to the technical scheme of the application, a plurality of brand robot instruction systems can be adapted, and the extensibility is good.

[0199] The functions described herein can be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions can be stored on or transferred over as one or more instructions or code on a computer-readable medium. Other examples and implementations are within the scope and spirit of the disclosure and appended claims. For example, due to the nature of software, functions described above can be implemented using software executed by a processor, hardware, firmware, hardwiring, or combinations of any of these. Also, each of the functions can be implemented as a separate function or combined as a single function in a process unit. Further, the functions can be implemented as separate units or combined as a single unit.

[0200] In several embodiments provided in the present application, it should be understood that the disclosed technical content can be implemented by other ways. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the units can be a logical function division, and another division mode can be used in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or modules shown or discussed can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0201] The units described as separate components can or can not be physically separate, and the components of the control device can or can not be physical units, that is, they can be located in one place or distributed on a plurality of units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0202] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the entire or part of the technical solutions that make contributions to the related art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0203] The above only describes the embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the scope of the claims of the present application.

Claims

1. A robot programming method, characterized in that: include: Receive task description statement; Parsing the received task description sentence through a pre-trained natural language processing model to extract action keywords, action parameters, and sequence identifiers; Constructing a semantic parsing tree based on the extracted action keywords, action parameters and sequence identifiers to determine the action sequence and logical dependency relationship; Converting the task description statement into a robot instruction according to the extracted action keywords and action parameters and the determined action sequence and logical dependency; The converted robot instructions are used to generate robot program codes executable by a target robot.

2. The method according to claim 1, characterized in that Also includes: Before generating an executable robot program of a target robot from the converted robot instructions, necessary parameters not involved in the task description statement are added to the converted robot instructions and / or parameter format conversion is performed according to preset grammatical rules.

3. The method according to claim 2, characterized in that The necessary parameters include: default parameters and / or context-derived parameters; The parameter format conversion includes: converting the position description in the task description statement into robot coordinates.

4. The method according to claim 1, wherein Also includes: During the generation phase of the robot program code, detecting in real time whether the generated code has conflicts; If conflicts are detected in the generated code, corrections are made according to the preset logic.

5. The method according to any one of claims 1 to 4, characterized in that Receive task description statements, including: A text box is provided in which a task description statement is received.

6. The method according to any one of claims 1 to 4, characterized in that Constructing a semantic parsing tree based on the extracted action keywords, action parameters, and sequence identifiers to determine the action sequence and logical dependency relationships, including: Create a node for each identified action, and create child nodes or attributes based on the corresponding action parameters and / or sequence identifiers to build a semantic parse tree to determine the order and logical dependencies.

7. A robot programming device, characterized in that: include: A receiving unit, configured to receive a task description statement; a parsing unit, configured to parse the task description sentence received by the receiving unit using a pre-trained natural language processing model, and extract action keywords, action parameters, and sequence identifiers; a determination unit, configured to construct a semantic parsing tree based on the action keywords, action parameters, and sequence identifiers extracted by the parsing unit, and determine the action sequence and logical dependency; a conversion unit, configured to convert the task description statement into a robot instruction according to the action keywords and action parameters extracted by the parsing unit and the action sequence and logical dependency determined by the determining unit; A generating unit is configured to generate a robot program code executable by a target robot from the robot instruction converted by the converting unit.

8. A storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A teaching pendant, characterized in that: The present invention comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented, or the programming device of the robot according to claim 7 is included.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the steps of the method according to any one of claims 1 to 6 when the computer program is executed by a processor.

Images