Robot grabbing self-adaptive method and system based on deep learning

Abstract

The invention discloses a robot grabbing self-adaptive method and system based on deep learning, and relates to the technical field of robot grabbing, and the method comprises the steps: constructing a digital twin model of an object grabbed by a robot; the master control system carries out data processing and analysis on the digital three-dimensional model; the control module receives an instruction of the master control system, an emergency grabbing option is started, and the robot is controlled to conduct self-adaptive grabbing on an object in an abnormal area; according to the robot, the digital three-dimensional model is established, the digital three-dimensional model is recognized and analyzed through the master control system, the result is transmitted to the control module, the robot can conduct self-adaptive grabbing on at least two stacked objects through the control module, meanwhile, data analysis at the robot end is achieved, edge calculation is conducted, and therefore the robot can grab more than two stacked objects in a self-adaptive mode. According to the method, the robot can realize self-adaptive grabbing in a more complex grabbing environment, so that the time delay of back-and-forth transmission of data is reduced, and the object grabbing accuracy of the robot is ensured.

Description

Technical Field

[0001] The present invention relates to the field of robot grasping technology, and in particular to a robot grasping adaptive method and system based on deep learning. Background Art

[0002] Robotic grasping technology is a core capability for realizing intelligent manufacturing, warehousing and logistics, and home service automation. Adaptive grasping methods, through a real-time perception-decision-execution closed loop, enable robots to autonomously adjust to environmental changes, and have become a research hotspot in recent years. A Chinese invention patent (CN119407773A) discloses a "robot visual grasping detection method, system, and medium for object stacking scenarios," specifically disclosing: extracting image features from the RGB image of objects in the work scene; determining an object detection frame and a grasping rectangle based on the image features, with the object detection frame representing the object category and position in the work scene, and the grasping rectangle representing the grasping posture of the object in the work scene; matching the object detection frame and the grasping rectangle; reflecting the matched object detection frame and grasping rectangle to the image features, and then performing operation relationship detection on the objects in the work scene to obtain an object operation relationship tree; the robot grasps the objects in the work scene based on the object operation relationship tree, object detection frame, and grasping rectangle. This enables accurate grasping of the robot in object stacking scenarios. The above technical solution achieves accurate grasping in the scene of stacked objects. However, it can only accurately grasp stacked objects, and the object grasping is performed under specific rules, which cannot adapt to more complex environments. For example, it can only accurately grasp a stack of at most two objects. Therefore, there is an urgent need for a deep learning-based robot grasping adaptive method and system to solve the above problems. Summary of the Invention

[0003] The purpose of the present invention is to provide a robot grasping adaptive method and system based on deep learning to solve the problems raised in the prior art.

[0004] To achieve the above objectives, the present invention provides the following technical solutions: a robot grasping adaptive method based on deep learning: using an RGB-D camera to capture and construct a digital three-dimensional model of the object grasped by the robot; The master control system uses a deep learning model to identify abnormal conditions of objects in the digital 3D model and determine whether emergency grasping conditions are triggered; The control module receives instructions from the master control system, activates the emergency grasping option, and controls the robot to adaptively grasp objects in abnormal states; The master control module is also provided with an external interface, which is used to connect to other control modules and robots through the external interface when the object is placed in an abnormal state, and cooperate with the original control module and robot to perform adaptive grasping of the object.

[0005] According to the above technical solution, for the training of the deep learning model, by collecting a dataset of images of non-abnormal objects and a dataset of images of real abnormal objects, and defining a loss function, the model is trained using the dataset and tested, and finally a deep learning model is obtained; The deep learning model identifies abnormal conditions and frames the stacked objects using a labeling tool to form an abnormal calibration frame; The marking tool is also used to mark any single object, with the marking point located at the physical center of the object to form a calibration point; For objects within the abnormal calibration frame, their three-dimensional data is extracted, and the contour lines of the objects within the abnormal calibration frame are extracted using image processing technology to obtain several groups of closed contour lines. These groups of closed contour lines are compared one by one with the contour lines of a single object for similarity, and the contour of the object on the top layer of the stacked objects is determined, and the top layer object is specially marked using annotation tools.

[0006] According to the above technical solution, the deep learning model sends the digital three-dimensional model containing abnormal calibration boxes, calibration points and special marks to the control module, which changes the robot's original grasping plan and sends the changed grasping plan to the grasping robot. The grasping robot executes the grasping instructions to complete the adaptive grasping of objects in abnormal conditions.

[0007] According to the above technical solution, for the abnormal state of the grasped object in the abnormal calibration frame, the grasping robot is used to grasp the object with special marks, and then the RGB-D camera installed on the grasping robot is used to collect three-dimensional data, and the collected three-dimensional data is transmitted back to the control module. The control module continues to perform image processing on the object in the abnormal calibration frame based on the three-dimensional data, and uses the annotation tool to specially mark the topmost object in the abnormal calibration frame until there is no stacking of objects in the abnormal calibration frame.

[0008] According to the above technical solution, the deep learning model also includes a data analysis unit, which is used to analyze other abnormal situations that occur during the object transportation process. The data analysis unit is connected to the object transportation control system and is used to receive relevant information data about the object transportation.

[0009] According to the above technical solution, the data analysis unit frames a first unit area in the digital three-dimensional model, counts the number of calibration points in the first unit area, and calculates the object density in the first unit area; When the data analysis unit receives the increase or decrease in the speed of the object conveyance, the data analysis unit continues to frame the second unit area of the digital three-dimensional model, counts the number of calibration points in the second unit area, and analyzes the density of the objects in the second unit area; When the speed of the object conveying increases, the first unit area is larger than the second unit area, and when the speed of the object conveying decreases, the first unit area is smaller than the second unit area; When the object density within the first unit area or the second unit area is greater than or equal to a set threshold, it is determined that other abnormal conditions have occurred.

[0010] According to the above technical solution, when other abnormal situations occur, additional control modules and grasping robots are connected through the external interface of the master control system, and the master control system sends data information to different control modules respectively, and grasps the objects transported on the conveyor belt through at least two grasping robots.

[0011] According to the above technical solution, at least two grasping robots are arranged in a grasping order. After the previous grasping robot completes grasping the object, it collects three-dimensional data of the conveyed object through the RGB-D camera and transmits the collected three-dimensional data to the control module of the next grasping robot. The next grasping robot plans a grasping plan for the conveyed object based on the latest collected three-dimensional data, and collects three-dimensional data again when the next grasping robot completes grasping. If this grasping robot is the last grasping robot arranged in sequence, the collected three-dimensional data will be transmitted to the control module of the first grasping robot arranged in sequence, so as to complete the entire grasping plan planning cycle at the control module end and realize adaptive grasping of objects.

[0012] A deep learning-based robotic grasping adaptive system, wherein the RGB-D camera used to build a digital three-dimensional model is located at the front end of the object conveying direction and directly above the object's spatial level, and the grasping robot is located in the middle of the conveying direction and behind the RGB-D camera; The control module includes a grasping planning unit and an instruction sending unit; The grasping planning unit is used to re-plan the grasping plan of the grasping robot according to the abnormal state of the object to be grasped when the emergency grasping option is enabled, and transmit the grasping plan to the instruction sending unit. The instruction sending unit is used to send the changed grasping plan to the grasping robot, and the grasping robot executes the instruction to adaptively grasp the object in the abnormal state.

[0013] According to the above technical solution, the grasping robot is also equipped with an RGB-D camera. After grasping an object, the grasping robot collects three-dimensional data through the RGB-D camera and transmits the collected three-dimensional data to the grasping planning unit for planning subsequent grasping plans on the control module side.

[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention establishes a digital three-dimensional model, identifies and analyzes the digital three-dimensional model through a master control system, and transmits the results to a control module. The control module can be used to enable the robot to adaptively grasp at least two or more stacked objects. At the same time, the present invention is also equipped with an external interface to connect more control modules and robots, so that the robot can also achieve adaptive grasping in more complex grasping environments. At the same time, the adaptability of the robot's adaptive grasping is improved. Moreover, for more complex grasping environments, data analysis and edge computing are implemented on the robot side, which reduces the delay in data transmission back and forth and ensures the accuracy of the robot's object grasping. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Schematic diagram of the process of the robot's adaptive grasping control of the present invention; Figure 2 Schematic diagram of a further process of the adaptive grasping control of the robot of the present invention; Figure 3 Schematic diagram of the relationship between the master control system and the control modules of the present invention; Figure 4 This is a further flow chart of the relationship between the overall monitoring system, the control module and the robot of the present invention. DETAILED DESCRIPTION

[0016] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0017] Example: Figure 1-Figure 2 As shown, the robot of this embodiment performs adaptive grasping of objects conveyed on a conveyor belt, specifically comprising the following steps: S1. Build a digital twin model of the object grasped by the robot; Specifically, the robot collects three-dimensional data of the object it grasps based on an RGB-D camera and constructs a digital three-dimensional model of the object; Furthermore, the RGB-D camera is located at the front end of the conveyor belt in the conveying direction and directly above the spatial level of the conveyor belt, and is used to collect three-dimensional data of objects passing through the RGB-D camera and construct a digital three-dimensional model. The robot is located in the middle of the conveyor belt in the conveying direction and behind the RGB-D camera, and is used to adaptively grasp objects on the conveyor belt; After the RGB-D camera acquires three-dimensional data, it completes the construction of the digital three-dimensional model through the process of point cloud registration, surface reconstruction, texture mapping and model optimization. Before the robot grasps an object, by building a digital three-dimensional model of the object to be grasped, on the one hand, it can plan the grasping plan in advance for the robot's adaptive grasping, and on the other hand, it can identify abnormal states of the object's placement so that the robot can take timely response measures and realize the robot's adaptive grasping.

[0018] S2, the master control system processes and analyzes the digital three-dimensional model; Specifically, the master control system identifies abnormal conditions of the constructed digital 3D model based on a deep learning model to determine whether emergency grasping conditions are triggered; The deep learning model is mainly used to identify and analyze the state of objects in the digital three-dimensional model, for example: objects pile up on the surface of the conveyor belt; For deep learning model training, we collect a dataset of images without abnormal objects and a dataset of images with real abnormal objects, define a loss function, train the model using the dataset, and complete the test to finally obtain a deep learning model. The deep learning model identifies abnormal conditions and frames objects in abnormal conditions using a labeling tool to form abnormal calibration frames; The marking tool is also used to mark any single object, with the marking point located at the physical center of the object to form a calibration point; Abnormal state refers to the phenomenon of objects stacking. The stacked objects are framed to facilitate the robot's adaptive grasping of the stacked objects. Any single object is marked to analyze and determine whether there are other abnormal situations in the transportation of objects on the conveyor belt, so that the robot can respond in time and perform adaptive grasping. For objects within the abnormal calibration frame, their 3D data is extracted, and the contour lines of the objects within the abnormal calibration frame are extracted using image processing technology to obtain several groups of closed contour lines. These groups of closed contour lines are compared one by one with the contour lines of the objects conveyed by the conveyor belt for similarity, and the contour of the object on the top layer of the stacked objects is determined and specially marked using annotation tools; For abnormal states where the grasped object has an abnormal calibration frame, a grasping robot is used to grasp the object with special marks, and then the RGB-D camera installed on the grasping robot is used to collect three-dimensional data, and the collected three-dimensional data is transmitted back to the control module. The control module continues to perform image processing on the object in the abnormal calibration frame based on the three-dimensional data, and uses the annotation tool to specially mark the topmost object in the abnormal calibration frame until there is no stacking of objects in the abnormal calibration frame.

[0019] The deep learning model also includes a data analysis unit for analyzing other abnormal conditions that may occur when objects are on the conveyor belt, such as an increase in the density of objects per unit area on the conveyor belt surface or an increase in the conveying speed of the conveyor belt for the objects. The design of the data analysis unit enables the optimal grasping solution to be adjusted for abnormal object conveying conditions in different scenarios, thereby enabling the grasping robot to adaptively grasp objects in different scenarios and improving the adaptability of the grasping robot.

[0020] Specifically, in this embodiment, the data analysis unit frames a unit area in the digital three-dimensional model, the unit area being S, and counts the number of calibration points in the unit area to determine the number of calibration points being N. The data analysis unit calculates the object density P within the unit area according to the formula P=N / S; When the object density P per unit area is greater than or equal to the set threshold, it is determined that the object density per unit area on the conveyor belt surface has increased and other abnormal conditions have occurred; The data analysis unit is connected to the conveyor control system and is used to receive the transmission speed data V of the conveyor. When the transmission speed V of the conveyor increases, the data analysis unit continues to frame the unit area of the digital three-dimensional model and analyze the object density P within the unit area. ’ , but the unit area of the frame is S ’ =a*S, where a is the proportional coefficient; As the transmission speed of the conveyor belt increases, in order to ensure that the density threshold remains unchanged and the data analysis results are not affected, the unit area framed in the digital three-dimensional model needs to be scaled proportionally to ensure the uniformity of density calculation and avoid setting multiple thresholds or modifying thresholds, which will increase the system calculation load. The proportional coefficient a is affected by the variation of the transmission speed of the transmission belt. When the variation of the transmission speed is greater than 0, a<1; and when the variation of the transmission speed is less than 0, a>1.

[0021] S3: The control module receives instructions from the master control system, activates the emergency grasping option, and controls the robot to adaptively grasp objects in abnormal areas; Specifically, such as Figure 2 As shown, the control module includes a grasping planning unit and an instruction sending unit; The grasping planning unit is used to re-plan the grasping plan of the robot according to the abnormal state of the object to be grasped when the emergency grasping option is enabled, and transmit the grasping plan to the instruction sending unit. The instruction sending unit sends the changed grasping plan to the robot, and the robot executes the instruction to adaptively grasp the object in the abnormal state; Furthermore, the robot's gripper is also equipped with an RGB-D camera for collecting three-dimensional data of the object before and after grasping. In this embodiment, when the abnormal situation is the stacking of objects, the grasping planning unit first grasps the objects with special marks in the abnormal calibration frame when planning the grasping plan. When grasping the specially marked objects, the RGB-D camera is used to collect three-dimensional data again, that is, point cloud data. The image processing technology is used on the robot side to extract the contour lines of the remaining objects in the abnormal calibration frame again, and several groups of closed contour lines are obtained again. The similarity of the several groups of closed contour lines is compared one by one with the contour lines of the objects transported by the conveyor belt to determine the contour of the object on the top layer among the remaining stacked objects. The robot then grasps the object on the top layer and repeats the above actions until there are no stacked objects in the abnormal calibration frame.

[0022] In this embodiment, the abnormal situation is when the density of objects per unit area on the surface of the conveyor belt increases or the conveying speed of the conveyor belt for the objects increases; When the density of objects per unit area is less than a set threshold, the grasping planning unit increases the robot's grasping speed when changing the grasping plan, thereby ensuring that the grasping function of the object is normally realized in the event of an abnormal situation; like Figure 3-Figure 4 As shown, the master control system is further provided with an external interface. When the density of objects per unit area is greater than or equal to a set threshold, an additional control module and a robot are connected via the external interface. At this time, the master control system sends data information to different control modules respectively, and at least two robots are used to grab objects transported on the conveyor belt to solve the situation where the density of objects per unit area on the conveyor belt is greater than or equal to the set threshold. Two or more robots are arranged on the side of the conveyor belt in the order of grasping. After the previous robot completes grasping the object, it will use the RGB-D camera to collect three-dimensional data of the object on the conveyor belt, that is, collect point cloud data, and transmit the collected three-dimensional data to the grasping planning unit of the control module of the next robot. The grasping planning unit of the next robot plans a grasping plan for the object on the conveyor belt based on the latest collected three-dimensional data, and when the next robot completes the grasping, it collects three-dimensional data again. If this robot is the last robot arranged in sequence, the collected three-dimensional data will be transmitted to the grasping planning unit of the first robot arranged in sequence, so as to complete the entire grasping plan planning cycle on the control module side. The amount of data transmission is reduced through edge computing, which makes the robot respond faster and grasp more accurately when grasping objects.

[0023] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A robot grasping adaptive method based on deep learning, characterized by: Use RGB-D cameras to capture and construct digital 3D models of objects grasped by the robot; The master control system uses a deep learning model to identify abnormal conditions of objects in the digital 3D model and determine whether emergency grasping conditions are triggered; The control module receives instructions from the master control system, activates the emergency grasping option, and controls the robot to adaptively grasp objects in abnormal states; The master control module is also provided with an external interface, which is used to connect to other control modules and robots through the external interface when the object is placed in an abnormal state, and cooperate with the original control module and robot to perform adaptive grasping of the object.

2. The robot grasping adaptive method based on deep learning according to claim 1 is characterized in that: For deep learning model training, we collect a dataset of images without abnormal objects and a dataset of images with real abnormal objects, define a loss function, train the model using the dataset, and complete the test to finally obtain a deep learning model. The deep learning model identifies abnormal conditions and frames the stacked objects using a labeling tool to form an abnormal calibration frame; The marking tool is also used to mark any single object, with the marking point located at the physical center of the object to form a calibration point; For objects within the abnormal calibration frame, their three-dimensional data is extracted, and the contour lines of the objects within the abnormal calibration frame are extracted using image processing technology to obtain several groups of closed contour lines. These groups of closed contour lines are compared one by one with the contour lines of a single object for similarity, and the contour of the object on the top layer of the stacked objects is determined, and the top layer object is specially marked using annotation tools.

3. The robot grasping adaptive method based on deep learning according to claim 2, characterized in that: The deep learning model sends the digital three-dimensional model containing the abnormal calibration frame, calibration points and special marks to the control module, which changes the robot's original grasping plan and sends the changed grasping plan to the grasping robot. The grasping robot executes the grasping instructions to complete the adaptive grasping of objects in abnormal conditions.

4. The robot grasping adaptive method based on deep learning according to claim 3 is characterized in that: For abnormal states where the grasped object has an abnormal calibration frame, a grasping robot is used to grasp the object with special marks, and then the RGB-D camera installed on the grasping robot is used to collect three-dimensional data, and the collected three-dimensional data is transmitted back to the control module. The control module continues to perform image processing on the object in the abnormal calibration frame based on the three-dimensional data, and uses the annotation tool to specially mark the topmost object in the abnormal calibration frame until there is no stacking of objects in the abnormal calibration frame.

5. The robot grasping adaptive method based on deep learning according to claim 2, characterized in that: The deep learning model also includes a data analysis unit, which is used to analyze other abnormal situations that occur during the object transportation process. The data analysis unit is connected to the object transportation control system and is used to receive relevant information data about the object transportation.

6. The deep learning-based robot grasping adaptive method according to claim 5, characterized in that: The data analysis unit frames a first unit area in the digital three-dimensional model, counts the number of calibration points in the first unit area, and calculates the object density in the first unit area; When the data analysis unit receives the increase or decrease in the speed of the object conveyance, the data analysis unit continues to frame the second unit area of the digital three-dimensional model, counts the number of calibration points in the second unit area, and analyzes the density of the objects in the second unit area; When the speed of the object conveying increases, the first unit area is larger than the second unit area, and when the speed of the object conveying decreases, the first unit area is smaller than the second unit area; When the object density within the first unit area or the second unit area is greater than or equal to a set threshold, it is determined that other abnormal conditions have occurred.

7. The robot grasping adaptive method based on deep learning according to claim 6, characterized in that: When other abnormal situations occur, additional control modules and grasping robots are connected through the external interface of the master control system. The master control system sends data information to different control modules respectively, and at least two grasping robots grasp the objects transported on the conveyor belt.

8. The robot grasping adaptive method based on deep learning according to claim 7, characterized in that: At least two grasping robots are arranged in a grasping order. After the previous grasping robot completes grasping the object, it uses an RGB-D camera to collect 3D data of the conveyed object and transmits the collected 3D data to the control module of the next grasping robot. The next grasping robot plans a grasping plan for the conveyed object based on the latest collected 3D data and collects 3D data again when the next grasping robot completes grasping. If this grasping robot is the last grasping robot arranged in sequence, the collected three-dimensional data will be transmitted to the control module of the first grasping robot arranged in sequence, so as to complete the entire grasping plan planning cycle at the control module end and realize adaptive grasping of objects.

9. A deep learning-based robot grasping adaptive system implementing the deep learning-based robot grasping adaptive method according to any one of claims 1 to 8, characterized in that: The RGB-D camera for establishing a digital three-dimensional model is located at the front end of the object conveying direction and directly above the object space level, and the grasping robot is located in the middle of the conveying direction and behind the RGB-D camera; The control module includes a grasping planning unit and an instruction sending unit; The grasping planning unit is used to re-plan the grasping plan of the grasping robot according to the abnormal state of the object to be grasped when the emergency grasping option is enabled, and transmit the grasping plan to the instruction sending unit. The instruction sending unit is used to send the changed grasping plan to the grasping robot, and the grasping robot executes the instruction to adaptively grasp the object in the abnormal state.

10. The deep learning-based robot grasping adaptive method and system according to claim 9, characterized in that: The grasping robot is also equipped with an RGB-D camera. After grasping an object, the grasping robot collects three-dimensional data through the RGB-D camera and transmits the collected three-dimensional data to the grasping planning unit for planning subsequent grasping plans on the control module side.

Images