Intelligent grabbing method of mechanical arm

Abstract

The invention relates to the technical field of mechanical arm control, and discloses an intelligent grabbing method of a mechanical arm. The method comprises the steps that the mechanical arm is powered on for self-inspection; acquiring an image of the task scene, and performing object recognition on the acquired image by using the multi-modal large model; constructing a task scene according to an identification result; inputting a natural language control instruction used for controlling the mechanical arm, analyzing the natural language control instruction by using the multi-modal large model, and planning a mechanical arm motion control sequence according to an analysis result so as to generate a structured data instruction used for controlling the mechanical arm; the structured data instruction is converted, a mechanical arm control service request is generated, and a to-be-grabbed target and a to-be-placed position or plane are determined according to the mechanical arm control service request; judging whether a to-be-captured target exists in the constructed task scene or not; and under the condition that the to-be-grabbed target exists, a grabbing task instruction and a placement task instruction are output to the mechanical arm, so that the mechanical arm is controlled to sequentially execute a grabbing action and a placement action.

Description

Technical Field

[0001] The present invention relates to the technical field of robot arm control, and in particular to an intelligent grasping method of a robot arm. Background Art

[0002] Robotic arms play an important role in the field of industrial automation and intelligent manufacturing. Robotic arms are deeply involved in many industrial production links. With the rapid development of artificial intelligence technology in recent years, people's demand for robotic arm automation and intelligence has continued to grow, in order to reduce production costs, improve production efficiency and product quality.

[0003] Traditional industrial robotic arms usually rely on pre-programmed instructions and fixed operating modes to perform tasks. They perform well in work environments that require repetitiveness and high precision. However, these robotic arms have limited flexibility and are relatively poorly adaptable to changing production needs and complex tasks. They often require human intervention to adjust the program to adapt to new production tasks or environmental changes, which not only increases operating costs, but also limits further improvements in production efficiency. Summary of the invention

[0004] The present invention provides a robot arm intelligent grasping method, which can solve the problems in the prior art.

[0005] The present invention provides a robot arm intelligent grasping method, wherein the method comprises:

[0006] The robot arm performs self-test after power-on;

[0007] Collect images of the mission scene and use a large multimodal model to perform object recognition on the collected images;

[0008] Build a task scenario based on the recognition results;

[0009] Input natural language control instructions for controlling the robotic arm, parse the natural language control instructions using a multimodal large model, and plan the robotic arm motion control sequence based on the parsing results to generate structured data instructions for controlling the robotic arm;

[0010] Convert the structured data instructions to generate a robot arm control service request, and determine the target to be grasped and the position or plane to be placed according to the robot arm control service request;

[0011] Determine whether the target to be grasped exists in the constructed task scenario;

[0012] When the target to be grasped exists, the grasping task instruction and the placing task instruction are output to the robot arm to control the robot arm to perform the grasping action and the placing action in sequence.

[0013] Preferably, before outputting the grab task instruction and the placement task instruction to the robot arm, the method further comprises:

[0014] Determine the gripper force of the robot arm.

[0015] Preferably, the control instructions for controlling the robotic arm are input in a natural language through voice or text.

[0016] Preferably, the method further comprises:

[0017] The objects in the constructed task scene are labeled and dynamically increased or decreased.

[0018] Preferably, the method further comprises:

[0019] Manage the positions and poses of all objects in the constructed task scene.

[0020] Preferably, managing the positions and postures of all objects in the constructed task scene includes:

[0021] The positions and postures of all objects in the constructed task scene are saved, displayed and deleted, and the object grasping and placing positions are determined.

[0022] Through the above technical solution, it is possible to grasp and place specific targets in the task scenario through natural language instructions, thereby achieving end-to-end control of the robotic arm, greatly simplifying the use process of the robotic arm and expanding the application scenarios of the robotic arm. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The included drawings are used to provide a further understanding of the embodiments of the present invention, which constitute a part of the specification, are used to illustrate the embodiments of the present invention, and together with the text description, explain the principles of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 A flowchart of a robot arm intelligent grasping method according to an embodiment of the present invention is shown;

[0025] Figure 2 A schematic diagram showing an example of intelligent grasping by a robotic arm according to an embodiment of the present invention is shown;

[0026] Figure 3 A schematic diagram of data flow according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0027] It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0029] Unless otherwise specifically stated, the relative arrangement of the parts and steps described in these embodiments, numerical expressions and numerical values ​​do not limit the scope of the present invention. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to the actual proportional relationship. The technology, method and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but in appropriate cases, the technology, method and equipment should be regarded as a part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0030] Figure 1 A flow chart of a robot arm intelligent grasping method according to an embodiment of the present invention is shown.

[0031] like Figure 1 As shown, an embodiment of the present invention provides a robot arm intelligent grasping method, wherein the method includes:

[0032] The robot arm performs a self-check after powering on, thus ensuring that the robot arm itself is in a normal state;

[0033] Collect images of the mission scene and use a large multimodal model to identify objects (targets) in the collected images;

[0034] Build a task scenario based on the recognition results;

[0035] Input the natural language control instructions for controlling the robot arm, use the multimodal large model to parse the natural language control instructions, and plan the robot arm motion control sequence according to the parsing results to generate structured data instructions for controlling the robot arm (generate serialized instructions). The data flow process is as follows: Figure 3 As shown;

[0036] Convert the structured data instructions to generate a robot arm control service request, and determine the target to be grasped and the position or plane to be placed according to the robot arm control service request;

[0037] Determine whether the target to be grasped exists in the constructed task scenario;

[0038] When the target to be grasped exists, the grasping task instruction and the placing task instruction are output to the robot arm to control the robot arm to perform the grasping action and the placing action in sequence.

[0039] After that, you can continue to send the next natural language control instruction for controlling the robotic arm or end the robotic arm control task.

[0040] Through the above technical solution, it is possible to grasp and place specific targets in the task scenario through natural language instructions, thereby achieving end-to-end control of the robotic arm, greatly simplifying the use process of the robotic arm and expanding the application scenarios of the robotic arm.

[0041] If the target to be grasped does not exist, the target to be grasped and the position or plane to be placed can be re-determined.

[0042] According to one embodiment of the present invention, before outputting the grasping task instruction and the placing task instruction to the robot arm, the method further includes:

[0043] Determine the gripper force of the robot arm.

[0044] Therefore, the target object can be grasped according to the determined gripper force, thereby achieving gripper force control.

[0045] According to one embodiment of the present invention, control instructions (natural language control instructions) for controlling the robot arm are input in a natural language manner through voice or text.

[0046] Specifically, the grasping method described in the present invention can adapt to a variety of open source or closed source large models, can be flexibly deployed online and offline, and builds a robotic arm control system based on the Robot Operating System (ROS) widely used in the robotics field, realizing the grasping and placement of specific targets through voice and text commands.

[0047] In addition, in terms of functionality, the present invention also designs a large model prompt word template for robot arm grasping, realizes the understanding of voice commands and text commands, task sequence planning and standardized data output, designs a communication interface that connects the robot arm motion control and the multimodal large model, and realizes the data conversion from control commands to ROS services (ROS Service); for robot arm planning and control, a functional interface with autonomous action planning, specified name target grasping, specified position placement, etc. is designed. Through the combination of these functions, end-to-end target grasping and placement functions through natural language or text commands are realized, simplifying the traditional robot arm control process.

[0048] According to one embodiment of the present invention, the method further includes:

[0049] The objects in the constructed task scene are labeled and dynamically increased or decreased.

[0050] That is, it is possible to establish a task scenario based on the multimodal large model scene recognition results, obtain information on all targets in the task scenario, and support dynamic addition and deletion of objects in the scene.

[0051] According to one embodiment of the present invention, the method further includes:

[0052] Manage the positions and poses of all objects in the constructed task scene.

[0053] Specifically, it can dynamically construct task scenarios based on perception results and manage the positions and postures of all objects in the scene.

[0054] According to an embodiment of the present invention, managing the positions and postures of all objects in the constructed task scene includes:

[0055] The positions and postures of all objects in the constructed task scene are saved, displayed and deleted, and the object grasping and placing positions are determined.

[0056] For example, the present invention can automatically plan the arrival path and target grabbing plan according to the specified target name; support specific posture grabbing, specified force grabbing, specified position placement, specified plane placement and other functions, such as Figure 2 shown.

[0057] It can be seen from the above embodiments that the robot arm intelligent grasping method described in the above embodiments of the present invention has at least the following advantages:

[0058] (1) End-to-end robotic arm motion control combined with multi-modal large model

[0059] The present invention provides an end-to-end robotic arm control method combined with a multimodal large model. The method can be widely adapted to a variety of open source / closed source large models, and has online and offline deployment capabilities. The method uses the large model's ability to understand and parse natural language to expand the robotic arm control method, and realizes end-to-end control of the robotic arm through human-like natural interaction methods such as voice and text. The method has broad application prospects.

[0060] (2) Automatic grasping and placement of robotic arms for natural language control

[0061] The present invention is a naturally interactive robotic arm control method, and designs control interfaces such as scene target labeling, dynamic increase and decrease of targets, automatic planning and grasping of the robotic arm, and adjustment of the gripper force. It realizes control modes such as grasping of designated targets, grasping of designated positions, placement of designated positions, and placement of designated planes. It greatly simplifies the robotic arm control process, lowers the threshold for robotic arm control, and provides strong support for the robotic arm to perform complex tasks.

[0062] In the description of the present invention, it is necessary to understand that the directions or positional relationships indicated by directional words such as "front, back, up, down, left, right", "lateral, vertical, perpendicular, horizontal" and "top, bottom" are usually based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description. Unless otherwise specified, these directional words do not indicate or imply that the devices or elements referred to must have a specific direction or be constructed and operated in a specific direction. Therefore, they cannot be understood as limiting the scope of protection of the present invention. The directional words "inside and outside" refer to the inside and outside relative to the contours of each component itself.

[0063] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used here to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" can include both "above" and "below". The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatially relative descriptions used here are interpreted accordingly.

[0064] In addition, it should be noted that the use of terms such as "first" and "second" to limit components is only for the convenience of distinguishing the corresponding components. If not otherwise stated, the above terms have no special meaning and therefore cannot be understood as limiting the scope of protection of the present invention.

[0065] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A robot arm intelligent grasping method, characterized in that: The method includes: The robot arm performs self-test after power-on; Collect images of the mission scene and use a large multimodal model to perform object recognition on the collected images; Build a task scenario based on the recognition results; Input natural language control instructions for controlling the robotic arm, parse the natural language control instructions using a multimodal large model, and plan the robotic arm motion control sequence based on the parsing results to generate structured data instructions for controlling the robotic arm; Convert the structured data instructions to generate a robot arm control service request, and determine the target to be grasped and the position or plane to be placed according to the robot arm control service request; Determine whether the target to be grasped exists in the constructed task scenario; When the target to be grasped exists, the grasping task instruction and the placing task instruction are output to the robot arm to control the robot arm to perform the grasping action and the placing action in sequence.

2. The method according to claim 1, characterized in that Before outputting the grasping task instruction and the placing task instruction to the robot arm, the method further includes: Determine the gripper force of the robot arm.

3. The method according to claim 2, characterized in that The control instructions for controlling the robot arm are input in a natural language through voice or text.

4. The method according to claim 3, characterized in that The method further includes: The objects in the constructed task scene are labeled and dynamically increased or decreased.

5. The method according to claim 4, characterized in that The method further includes: Manage the positions and poses of all objects in the constructed task scene.

6. The method according to claim 5, characterized in that Managing the poses of all objects in the constructed mission scene includes: The positions and postures of all objects in the constructed task scene are saved, displayed and deleted, and the object grasping and placing positions are determined.

Images