A natural language-based robot instruction operation method, system and medium

Abstract

The present invention relates to a natural language-based robot instruction operation method, system, and medium, the method comprising the following steps: S1 defining the action primitive labels required by the robot; S2 acquiring visual image information in the robot operating environment and language instructions given by the operator in natural language; S3 inputting the visual image information and the operator's language instructions into a language-visual target mask model, outputting an action primitive target mask map; S4 inputting the visual image information and the action primitive target mask map into a reinforcement learning-based policy network, generating an action primitive operation instruction tuple corresponding to the corresponding operation target; S5 parsing the acquired action primitive operation instruction tuple, converting it into the robot joint target position, and controlling the robot to execute the action primitive. The method can enable the robot to complete the task objectives indicated by different language instructions, master a variety of operation skills, and adapt to the working environment of multiple task objectives.

Description

Technical Field

[0001] The present invention relates to the field of intelligent robot control technology, and in particular to a natural language-based robot instruction operation method, system and medium. Background Art

[0002] Robotics has been widely used in various fields, including industrial manufacturing and logistics, replacing humans in many repetitive and high-intensity tasks and further improving productivity. However, traditional robotic applications focus on structured scenarios with a single task objective, and the robot's operational skills are fixed. Changes in the task objective or complex operational scenarios require changes to the entire robot's control method, resulting in a lack of flexibility and autonomy.

[0003] Currently, AI-based robotics primarily utilizes imitation learning and deep reinforcement learning. While these approaches can enable robots to adapt to complex operational scenarios, most research focuses on directly manipulating robots end-to-end, addressing the operational challenges of completing a single task. These strategies often lack generalizability, requiring retraining for new tasks and making them incapable of adapting to multi-task environments.

[0004] In real life, natural language is an important way for humans to communicate information. Humans can flexibly express a variety of task goals through natural language. Intelligent robots should have the ability to semantically understand language instructions represented in natural language form, infer the corresponding operation goals, and perform corresponding action planning to complete the task goals indicated by different language instructions. Summary of the Invention

[0005] In response to the technical problems existing in the prior art, one of the purposes of the present invention is to provide a robot instruction operation method based on natural language, so as to enable the robot to complete the task objectives indicated by instructions in different languages, master a variety of operating skills, and adapt to the working environment of multiple task objectives.

[0006] A second object of the present invention is to provide a robot instruction operating system based on natural language.

[0007] A third object of the present invention is to provide a storage medium.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] In a first aspect, the present invention provides a natural language-based robot instruction operation method, the method comprising the following steps:

[0010] S1. Definition of robot action primitive library: According to the working environment, define the action primitive labels required by the robot, which can be expressed as:

[0011] Action Primitives={a0(0,x0),a1(1,x1),···,a i (i,x i ),···,a n (n,x n )};

[0012] Among them, the name of the i-th motion primitive is a i , the corresponding label value is i, and the corresponding operation instruction tuple is x i ;

[0013] S2, Environmental Information Perception: Acquiring visual image information in the robot's operating environment and language instructions given by the operator in natural language;

[0014] S3, Robot Task Understanding: Input visual image information and operator’s language instructions into the language-visual target mask model and output action primitive target mask map;

[0015] The action primitive target mask image has the same resolution as the input visual image information, and the pixel value corresponding to the operation target indicated by the language instruction is the action primitive label value that the robot should perform on the operation target, and the action primitive label value is defined in step S1;

[0016] S4, generating robot operation instructions: inputting the visual image information obtained in step S2 and the action primitive target mask map output in step S3 into the reinforcement learning-based policy network to generate an operation instruction tuple of the corresponding action primitive for the corresponding operation target, wherein the specific parameters of the operation instruction tuple are defined by step S1;

[0017] S5. Robot bottom-level motion control: The action primitive operation instruction tuple obtained in step S4 is further parsed and converted into the target position of the robot joint through methods such as coordinate system transformation and inverse kinematics solution, so as to control the robot to move and execute the action primitive.

[0018] Furthermore, the operation instruction tuple described in step S1 includes parameters such as the specific operation position and operation posture of the robot end.

[0019] Furthermore, the visual image information in step S2 includes an RGB image and a depth image; and the language instruction in step S2 specifies robot action attributes and object attributes.

[0020] Furthermore, the language instruction in step S2 follows an instruction template in a certain format.

[0021] Furthermore, the language-visual target mask model described in step S3 includes a visual feature encoding module, a language feature encoding module, a multimodal feature fusion module, and a feature decoding module. The feature representation of the image information is obtained based on the visual feature encoding module, and the feature representation of the language instruction is obtained based on the language feature encoding module. The multimodal fusion module multimodally fuses the feature representation of the image information with the feature representation of the language instruction to obtain a fused feature representation. The feature decoding module upsamples the fused feature representation to finally obtain an action primitive target mask map.

[0022] Furthermore, the working process of the multimodal feature fusion module specifically includes the following sub-steps:

[0023] S31: Given a visual feature V i With language feature L.

[0024] S32: Pass the language feature L through two fully connected layers, and further copy and expand the generated feature vector to generate the multiplication factor α i and bias parameter β i , multiplication factor α i and bias parameter β i Dimensions and visual features of V i same;

[0025] S33: Combine visual features with multiplication factor α i Perform element-by-element multiplication with the bias parameter β i Add element by element to generate fusion feature G i , which can be expressed as follows:

[0026]

[0027] Furthermore, the reinforcement learning-based policy network described in step S4 is constructed and trained using a deep Q-network approach, and the process of generating the operation instruction includes the following steps:

[0028] S41: Given visual image information and action primitive target mask map, input into the deep Q network;

[0029] S42: The deep Q network generates an action primitive Q value map end-to-end. The action primitive Q value map has the same resolution as the input visual image information and the action primitive target mask map. The number of channels is the same as the number of corresponding action primitive operation postures. The pixel value Q at a certain point on the action primitive Q value map is i (u i ,v i ,r i ) represents: the robot is at the pixel point (u i ,vi ) on channel r i The action Q value of the corresponding operation posture execution action primitive;

[0030] S43: Take the point Q with the largest pixel value on the action primitive Q value graph max (u max ,v max ,r max ) as the optimal execution point of the action primitive, that is, the optimal operation position of the action primitive is the pixel point (u max ,v max ), the best operating posture is channel r max Corresponding operating posture.

[0031] In a second aspect, the present invention further provides a natural language-based robot instruction operating system, comprising a visual information acquisition module, a computer control module, and a robot control module;

[0032] The visual information acquisition module is used to acquire visual image information in real time;

[0033] The computer control module includes a human-computer interaction submodule and a calculation reasoning submodule;

[0034] The human-computer interaction submodule is used to observe the acquired visual image information in real time and obtain the language instructions input by the operator;

[0035] The computational reasoning submodule is used to store and run the language-visual target mask model and the reinforcement learning-based policy network;

[0036] The robot control module is used for the bottom-level motion control of the robot, including coordinate system conversion of the operation position, operation posture, etc. in the operation instructions, and performing inverse kinematics solution to drive the joint motor of the external robot to reach the target position.

[0037] Furthermore, the visual image information perception module obtains visual image information through an external RGB-D camera and returns it to the computer control module.

[0038] In a third aspect, the present invention further provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned natural language-based robot instruction operation method.

[0039] The present invention has the following beneficial effects:

[0040] Traditional robotic applications focus on structured scenarios with single mission objectives and single operational skills, and are unable to adapt to multi-mission environments. Existing AI-based robotic operation methods, while enabling robots to autonomously master operational skills, are typically used to address operational challenges with a single mission objective. The method presented in this paper allows robots to flexibly change operational objectives based on verbal instructions from the operator, inferring which object should be executed by which action primitive. Through reinforcement methods, the robot acquires a wider range of operational skills, achieving greater flexibility and autonomy, and being able to complete mission objectives represented by different verbal instructions, making it suitable for multi-mission environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a schematic diagram of the workflow of a natural language-based robot instruction operation method described in the present invention.

[0042] Figure 2 A schematic diagram of an action primitive target mask diagram according to the method of the present invention.

[0043] Figure 3 Schematic diagram of the structure of the language-visual target mask model described in the method of the present invention.

[0044] Figure 4 A schematic diagram of the overall working process of a natural language-based robot instruction operation method provided in an embodiment of the present invention.

[0045] Figure 5 This is a schematic structural diagram of a natural language-based robot instruction operating system described in the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described in detail below with reference to the embodiments and the accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0047] like Figure 1 As shown, the present invention discloses a robot instruction operation method based on natural language. Taking the robot plane grasping and placing operation environment as an example, its workflow is described as follows:

[0048] S1 Action primitive library definition: According to the working environment, define the action primitive labels required by the robot, which can be expressed as:

[0049] Action Primitives={grasp(1,(x grasp ,y grasp ,γ grasp )),place(2,(x place ,y place )),}

[0050] Among them, the grab action primitive grasp(1,(x grasp ,y grasp ,γ grasp )), which represents the label value of the grab action primitive is 1, and the operation instruction tuple includes the two-dimensional coordinates (x grasp ,y grasp )’s grasping operation position, grasping operation posture γ grasp The rotation angle perpendicular to the working plane includes:

[0051] γ grasp ={0:0°1:30°2:60°3:90°4:-30°5:-60°};

[0052] Among them, 0 represents a rotation angle of 0°, 1 represents a rotation angle of 1°, and so on.

[0053] Also includes the place action primitive place(2,(x place ,y place )), which represents the label value of the grab action primitive is 2, and the operation instruction tuple includes the two-dimensional coordinates (x grasp ,y grasp ) placement operation position,

[0054] S2 Environmental Information Perception: Acquires visual image information in the robot's operating environment, as well as language instructions given by the operator in natural language, such as Figure 4 As shown in step S2.

[0055] The visual image information includes an RGB image and a depth image, and the resolution is H×W.

[0056] The language instructions specify the robot's action attributes and object attributes, and follow a certain format of instruction templates including:

[0057] Stack{object1}on the{object2};

[0058] Pick the{object1};

[0059] Put the{object1}into the{object2};

[0060] Among them, stack / pick / put are the action attributes of the robot, and object1 / object2 are the object attributes that the robot needs to operate. The object attributes can include object category, shape, color, and position, such as language instructions:

[0061] Put the rightmost trapezoid into the top plate;

[0062] Rightmost / top is the position attribute of the object, and trapezoid / plate is the shape attribute of the object.

[0063] S3 Robot task understanding: Input the RGB image obtained in step S2 and the operator's language instructions into the language-vision target mask model, and output the action primitive target mask map.

[0064] Among them, such as Figure 3 As shown, the language-visual target mask model described in S3 includes a visual feature encoding module, a language feature encoding module, a multimodal feature fusion module, and a feature decoding module. The feature representation of the image information is obtained based on the visual feature encoding module, and the feature representation of the language instruction is obtained based on the language feature encoding module. The multimodal fusion module multimodally fuses the feature representation of the image information with the feature representation of the language instruction to obtain a fused feature representation. The feature decoding module upsamples the fused feature representation to finally obtain an action primitive target mask map.

[0065] Furthermore, the working process of the multimodal feature fusion module specifically includes the following sub-steps:

[0066] S31: Given a visual feature V i and language feature L;

[0067] S32: Pass the language feature L through two fully connected layers, and further copy and expand the generated feature vector to generate the multiplication factor α i and bias parameter β i , multiplication factor α i and bias parameter β i Dimensions and visual features of V i same;

[0068] S33: Combine visual features with multiplication factor α i Perform element-by-element multiplication with the bias parameter β i Add element by element to generate fusion feature G i , which can be expressed as follows:

[0069]

[0070] Among them, the obtained action primitive target mask image has the same resolution as the input RGB image, and the pixel value corresponding to the operation target indicated by the language instruction is the action primitive label value that the robot should perform on the operation target. For example, under the language instruction of Put the rightmost trapezoid into the top plate, the pixel value of the area where the rightmost trapezoid is located in the action primitive target mask image is 1, indicating that the rightmost trapezoid should be grasped, and the pixel value of the area where the top plate is located is 2, indicating that the top plate should be placed. Figure 2 shown.

[0071] S4 Robot operation instruction generation: The visual image information obtained in step S2 and the action primitive target mask map output in step S3 are input into the reinforcement learning-based policy network to generate an operation instruction tuple of the corresponding action primitive of the corresponding operation target. The specific parameters of the operation instruction tuple are defined by step S1.

[0072] Among them, such as Figure 4 As shown in step S4, the reinforcement learning-based policy network is constructed and trained using a deep Q network. The process of generating the operation instruction includes the following steps:

[0073] S41: Given an RGB image, a depth image, and an action primitive target mask, input them into the deep Q network;

[0074] S42: The deep Q network generates the grab Q value map and the placement Q value map end to end. The resolution of the grab Q value map and the placement Q value map is also H×W. The number of channels of the grab Q value map is related to the grasping operation posture γ grasp The number of channels is the same, that is, 6 channels; the placement operation only needs to consider the placement operation position, so the number of channels for placing the Q value graph is 1.

[0075] Grab the pixel value Q of a certain point on the Q value graph i (u i ,v i ,r i ) represents: the robot is at the pixel point (u i ,v i ) on channel r i Corresponding operating posture γ grasp (r i ) is the action Q value for the grasping action. For example, Q(100,100,1), the robot's operating position should be the position of the pixel point (100,100), and the rotation angle perpendicular to the working plane should be the rotation angle γ corresponding to channel 1 grasp (1)=30°.

[0076] S43: Take the point Q with the largest pixel value on the Q value graph g (u g ,v g ,r g ) as the optimal execution point for the grabbing operation, that is, the optimal operation position for the grabbing operation is the pixel point (u g ,v g ), the best operating posture is channel r g Corresponding operating posture γ grasp (r g ).

[0077] Take the point Q with the largest pixel value on the Q value map p (u p ,v p ) as the optimal execution point for the placement operation, that is, the optimal operation position for the placement operation is the pixel point (u p ,v p ).

[0078] In summary, the output grab operation instruction grab(u g ,v g ,γ grasp (r g )), place operation instruction place(u p ,v p ).

[0079] S5 Robot bottom motion control: further analyze the action primitive operation instruction tuple obtained in step S4, and convert it into the coordinates in the robot coordinate system through the conversion between the robot coordinate system and the pixel coordinate system. For example, the grasping operation instruction grasp(u g ,v g ,γ grasp (r g )) is converted to the coordinates of the robot coordinate system grasp(x g ,y g ), the robot end posture is perpendicular to the working plane rotation γ grasp (r g ), and then the target position of each joint of the robot is obtained through inverse kinematics solution, and the joint motor is driven to move to complete the grasping operation.

[0080] For the placement operation instruction place(u p ,v p ), the above process is also executed, the robot movement process is as follows Figure 4 As shown in step S5.

[0081] In addition, if Figure 5As shown, the present invention also provides a natural language-based robot instruction operating system, including a visual information acquisition module, a computer control module, and a robot control module.

[0082] The visual information acquisition module acquires RGB images and depth images in real time through an external RGB-D camera and returns them to the computer control module.

[0083] The computer control module includes a human-computer interaction submodule and a calculation reasoning submodule.

[0084] The human-computer interaction submodule will acquire visual image information from the visual information acquisition module in real time and display it to the operator in real time. At the same time, the operator can input language instructions through the human-computer interaction submodule and convey control commands on whether the robot executes the language instructions.

[0085] The computational reasoning submodule is used to store and run the language-visual target mask model and the reinforcement learning-based policy network. When the human-machine interaction submodule transmits a control command for the robot to execute a language instruction, the computational reasoning submodule obtains the operator's language instruction from the human-machine interaction submodule and obtains the current visual image information from the visual information acquisition module. It then sequentially runs the language-visual target mask model and the reinforcement learning-based policy network to obtain the robot's operating instructions and transmits them to the robot control module.

[0086] The robot control module obtains the robot's operating instructions from the calculation and reasoning sub-module, performs coordinate system conversion on the operating position, operating posture, etc. in the operating instructions, and performs inverse kinematics solution to obtain the target position of each joint of the external robot, and drives the joint motor of the external robot to reach the target position, thereby controlling the robot to complete the corresponding operation.

[0087] In addition, the present invention also provides a storage medium having computer instructions or programs that can execute a natural language-based robot instruction operation provided by the present invention. When the computer instructions or program are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method can be obtained.

[0088] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A robot instruction operation method based on natural language, characterized in that: The following steps are involved: S1. Definition of robot action primitive library: According to the working environment, define the action primitive labels required by the robot, which can be expressed as: Action Primitives={a0(0,x0),a1(1,x1),···,a i (i,x i ),···,a n (n,x n )}; Among them, the name of the i-th motion primitive is a i , the corresponding label value is i, and the corresponding operation instruction tuple is x i ; S2, Environmental Information Perception: Acquiring visual image information in the robot's operating environment and language instructions given by the operator in natural language; S3, Robot Task Understanding: Input visual image information and operator’s language instructions into the language-visual target mask model and output action primitive target mask map; The action primitive target mask image has the same resolution as the input visual image information, and the pixel value corresponding to the operation target indicated by the language instruction is the action primitive label value that the robot should perform on the operation target, and the action primitive label value is defined in step S1; S4, generating robot operation instructions: inputting the visual image information obtained in step S2 and the action primitive target mask map output in step S3 into the reinforcement learning-based policy network to generate an operation instruction tuple of the corresponding action primitive for the corresponding operation target, wherein the specific parameters of the operation instruction tuple are defined by step S1; S5. Robot bottom-level motion control: The action primitive operation instruction tuple obtained in step S4 is further parsed and converted into the target position of the robot joint through coordinate system transformation and inverse kinematics solution method, so as to control the robot to move and execute the action primitive.

2. The natural language-based robot instruction operation method according to claim 1, characterized in that: The operation instruction tuple described in step S1 includes the specific operation position and operation posture parameters of the robot end.

3. The natural language-based robot instruction operation method according to claim 1, characterized in that: The visual image information described in step S2 includes an RGB image and a depth image; the language instruction described in step S2 specifies the robot action attributes and object attributes.

4. The natural language-based robot instruction operation method according to claim 3, characterized in that: The language instruction in step S2 follows an instruction template of a certain format.

5. The natural language-based robot instruction operation method according to claim 1, characterized in that: The language-visual target mask model described in step S3 includes a visual feature encoding module, a language feature encoding module, a multimodal feature fusion module and a feature decoding module. The feature representation of the image information is obtained based on the visual feature encoding module, and the feature representation of the language instruction is obtained based on the language feature encoding module. The multimodal fusion module performs multimodal fusion on the feature representation of the image information and the feature representation of the language instruction to obtain a fused feature representation. The feature decoding module upsamples the fused feature representation to finally obtain the action primitive target mask map.

6. The method for operating a robot command based on natural language according to claim 5, characterized in that: The working process of the multimodal feature fusion module specifically includes the following sub-steps: S31: Given a visual feature V i and language feature L; S32: Pass the language feature L through two fully connected layers, and further copy and expand the generated feature vector to generate the multiplication factor α i and bias parameter β i , multiplication factor α i and bias parameter β i Dimensions and visual features of V i same; S33: Combine visual features with multiplication factor α i Perform element-by-element multiplication with the bias parameter β i Add element by element to generate fusion feature G i , which can be expressed as follows:

7. The natural language-based robot instruction operation method according to claim 5, characterized in that: The reinforcement learning-based policy network described in step S4 is constructed and trained using a deep Q-network approach. The process of generating the operation instructions includes the following steps: S41: Given visual image information and action primitive target mask map, input into the deep Q network; S42: The deep Q network generates an action primitive Q value map end-to-end. The action primitive Q value map has the same resolution as the input visual image information and the action primitive target mask map. The number of channels is the same as the number of corresponding action primitive operation postures. The pixel value Q at a certain point on the action primitive Q value map is i (u i ,v i ,r i ) represents: the robot is at the pixel point (u i ,v i ) on channel r i The action Q value of the corresponding operation posture execution action primitive; S43: Take the point Q with the largest pixel value on the action primitive Q value graph max (u max ,v max ,r max ) as the optimal execution point of the action primitive, that is, the optimal operation position of the action primitive is the pixel point (u max ,v max ), the best operating posture is channel r max Corresponding operating posture.

8. The operating system for a natural language-based robot instruction operation method according to any one of claims 1 to 7, characterized in that: It includes visual information acquisition module, computer control module and robot control module; The visual information acquisition module is used to acquire visual image information in real time; The computer control module includes a human-computer interaction submodule and a calculation reasoning submodule; The human-computer interaction submodule is used to observe the acquired visual image information in real time and obtain the language instructions input by the operator; The computational reasoning submodule is used to store and run the language-visual target mask model and the reinforcement learning-based policy network; The robot control module is used for the bottom-level motion control of the robot, including coordinate system conversion of the operation position and operation posture in the operation instruction, and performing inverse kinematics solution to drive the joint motor of the external robot to reach the target position.

9. The operating system according to claim 8, wherein: The visual image information perception module obtains visual image information through an external RGB-D camera and returns it to the computer control module.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the natural language-based robot instruction operation method described in any one of claims 1 to 7 is implemented.

Images