Visual system applied to destacking and stacking of collaborative robot

Abstract

The invention belongs to the technical field of artificial intelligence, and discloses a visual system applied to stacking and unstacking of a collaborative robot. The visual system comprises an image acquisition module, a communication module and a visual processing module integrated in a robot control system. Three different types of cameras are integrated in the image acquisition module, and the image acquisition module is used for calling a corresponding camera to acquire an image according to the camera type in the camera connection instruction; the image acquisition module is in communication connection with the visual processing module through the communication module; the visual processing module comprises a hand-eye calibration unit, a positioning unit and an automatic deviation correction unit; wherein the hand-eye calibration unit is used for calibrating the pose of the robot according to an image of a mechanical arm of the robot; the positioning unit is used for calculating point location data of the target object according to the image of the to-be-grabbed material; and the automatic correction unit is used for correcting the point location data. By integrating a visual system and a robot control system, the operation complexity of the system is reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a visual system used for collaborative robot palletizing. Background Art

[0002] In the current field of industrial automation, especially in robot palletizing and depalletizing operations, the integration of vision systems has become one of the key technologies to improve robot performance and efficiency. However, although existing technologies provide a certain degree of automation and visual assistance, there are still the following significant defects: In existing systems, vision systems and robot control systems are often operated as independent modules, requiring complex manual configuration and multi-step operations to achieve coordination between systems, which greatly increases the complexity of system operation. At the same time, because the vision system and robot control system need to be operated independently, the traditional hand-eye calibration process usually requires manual intervention, which requires high skills of operators and limits the applicability and rapid deployment of the system. Summary of the invention

[0003] To this end, an embodiment of the present application provides a visual system applied to collaborative robot depalletizing and stacking, which reduces the operational complexity of the robot system.

[0004] This application is achieved through the following technical solutions:

[0005] A visual system for collaborative robot palletizing and unpalletizing, comprising:

[0006] An image acquisition module, a communication module and a visual processing module, wherein the visual processing module is integrated in the robot control system;

[0007] The image acquisition module is used to call the corresponding camera to acquire images according to the camera type in the camera connection instruction, and three different types of cameras are integrated in the image acquisition module;

[0008] A communication module, used to establish a communication connection between the image acquisition module and the visual processing module;

[0009] The visual processing module includes a hand-eye calibration unit, a positioning unit, and an automatic deviation correction unit;

[0010] A hand-eye calibration unit, used to calibrate the position and posture of the robot according to the image of the robot arm acquired by the image acquisition module;

[0011] A positioning unit, used to calculate the point data of the target object based on the image of the material to be grasped acquired by the image acquisition module;

[0012] The automatic deviation correction unit is used to correct the point data so that the robot's suction cup can pick up one material at a time.

[0013] In a preferred example of the present application, it can be further configured that the cameras integrated in the image acquisition module include: a binocular camera, a structured light camera and an infrared camera.

[0014] In a preferred example of the present application, it can be further configured that the image acquisition module is also used to manage three different types of cameras through C++ code factory mode and strategy mode.

[0015] In a preferred example of the present application, it can be further configured that the system further includes:

[0016] The visual feedback unit is used to monitor the real-time grasping path of the robot arm in real time during the robot grasping operation, and adjust the robot's operation strategy according to the real-time grasping path.

[0017] In a preferred example of the present application, it can be further configured that the hand-eye calibration unit is also used for:

[0018] Sending a hand-eye calibration instruction to the robot control system so that the robot control system moves the robot's mechanical arm to a preset position according to the hand-eye calibration instruction;

[0019] Sending a hand-eye calibration instruction to the image acquisition module, so that the image acquisition module controls the camera to acquire image information of the robot's mechanical arm at a preset position according to the hand-eye calibration instruction;

[0020] An image of the robotic arm at a preset position is received, and a relative position relationship between the camera and the robot is calculated based on the image to achieve hand-eye calibration between the camera and the robot.

[0021] In a preferred example of the present application, it can be further configured that the hand-eye calibration unit interacts with the robot control system by calling the SDK.

[0022] In a preferred example of the present application, it can be further configured that the positioning unit is also used to: convert the image of the target object into point cloud data using an Open3D image processing algorithm;

[0023] Preprocessing the point cloud data to obtain preprocessed point cloud data;

[0024] The pre-processed point cloud data is clustered and identified by a clustering algorithm to obtain a clustering result;

[0025] The center point of the target object is extracted from the clustering result as the point data of the target object.

[0026] In a preferred example of the present application, it can be further configured that the visual feedback unit is used to: when the real-time grasping path of the robot manipulator arm is greater than a preset threshold of the robot arm, split the grasping path corresponding to the current point data into multiple transition paths, and feed back the multiple transition paths to the robot control system.

[0027] In a preferred example of the present application, it can be further configured that the visual processing module is connected to the image acquisition module via TCP / IP.

[0028] In a preferred example of the present application, it can be further configured that the system also includes a user interaction interface, and the user interaction interface is used to display the operation status, real-time image information, error information and operation guidance information during the robot grasping operation.

[0029] In summary, compared with the prior art, the technical solution provided in the embodiment of the present application has at least the following beneficial effects:

[0030] The vision system of the present application includes an image acquisition module, a communication module and a vision processing module, wherein the vision processing module is integrated in a robot control system; the image acquisition module is used to call a corresponding camera to acquire images according to a camera type in a camera connection instruction, and three different types of cameras are integrated in the image acquisition module; the communication module is used to establish a communication connection between the image acquisition module and the vision processing module; the vision processing module includes a hand-eye calibration unit, a positioning unit and an automatic deviation correction unit; the hand-eye calibration unit is used to calibrate the posture of the robot according to an image of the robot manipulator acquired by the image acquisition module; the positioning unit is used to calculate the point data of the target object according to the image of the material to be grasped acquired by the image acquisition module; the automatic deviation correction unit is used to correct the point data so that the robot's suction cup can absorb one material at a time. The present application integrates the vision processing module into the robot control system for depalletizing and palletizing operations, so that users can realize the integrated operation of calibration and depalletizing without complicated configuration. The integration greatly simplifies the use process of the robot vision system and improves the operating efficiency. The seamless coordination between the vision system and the robot control system can significantly improve the overall operating efficiency of the system and reduce delays and errors caused by the lack of coordination between systems. The modular design of software and hardware enables the system to flexibly adapt to different industrial environments and operational requirements. Users can choose the appropriate module combination according to specific needs, which is easy to upgrade and maintain. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of the structure of a visual system for collaborative robot palletizing provided in one embodiment of the present application;

[0032] Figure 2A schematic diagram of the structure of a visual system applied to collaborative robot palletizing and unpalletizing provided in another embodiment of the present application;

[0033] Figure 3 A schematic diagram of the structure of a visual system for collaborative robot palletizing provided in another embodiment of the present application;

[0034] Description of reference numerals:

[0035] 1-image acquisition module, 2-communication module, 3-visual processing module, 31-hand-eye calibration unit, 32-positioning unit, 33-automatic correction unit, 34-visual feedback unit, 4-user interaction interface. DETAILED DESCRIPTION

[0036] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, those skilled in the art may make modifications to the present embodiment without any creative contribution as needed, but such modifications are protected by the patent law as long as they are within the scope of the claims of the present application.

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0038] In addition, the term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application, unless otherwise specified, generally indicates that the associated objects before and after are in an "or" relationship.

[0039] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with basically the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on quantity and execution order.

[0040] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. Figure 1 As shown, the visual system proposed in this application for collaborative robot palletizing includes:

[0041] Image acquisition module 1, communication module 2 and visual processing module 3, wherein the visual processing module 3 is integrated in the robot control system. It should be noted that the robot control system is the core part of the collaborative robot, which is responsible for receiving input signals, processing information, and controlling the movement and execution of tasks of the robot. The visual processing module 3 is encapsulated in an independent unit and integrated into the robot control system in a modular form, which means that the visual processing module 3 can be inserted into the robot control system as an independent component without large-scale modification or redesign of other parts of the robot control system. By directly integrating the visual system function into the robot's control software, by simplifying the system architecture and reducing the complexity of operation, it not only improves the user experience, but also enhances the stability and reliability of the system, thereby reducing maintenance costs. This design improves the practicality and economy of the robot system.

[0042] The visual processing module 3 integrated in the robot control system is connected to the image acquisition module 1 through the communication module 2. Specifically, it communicates with the camera in the image acquisition module 1 through TCP / IP. In the actual implementation process, the camera in the image acquisition module 1 is configured as a node in the TCP / IP network, and a static IP address is assigned to each camera to ensure that the network interface of the camera and the robot control system are in the same network or subnet so that they can communicate with each other.

[0043] The image acquisition module 1 is used to call the corresponding camera to capture images according to the camera type in the camera connection instruction, wherein the image acquisition module integrates three different types of cameras, the first camera, the second camera and the third camera. When an image acquisition task is required, the visual processing module 3 will send a camera connection instruction, which carries the camera configuration information that needs to be called and communicated, and the camera configuration information includes but is not limited to the IP and port number. Once the camera configuration is completed and the communication connection is established, the visual processing module 3 will send a capture command to the camera, and the camera will then start capturing images and transmit the captured images to the visual processing module 3 via the communication module 2 in TCP / IP mode.

[0044] The visual processing module 3 includes a hand-eye calibration unit 31, a positioning unit 32 and an automatic deviation correction unit 33;

[0045] The hand-eye calibration unit 31 is used to calibrate the position and posture of the robot according to the image of the robot arm acquired by the image acquisition module 1;

[0046] The positioning unit 32 is used to calculate the point data of the target object according to the image of the material to be grasped acquired by the image acquisition module 1; the target object is the object determined to be grasped in the material to be grasped.

[0047] The automatic deviation correction unit 33 is used to correct the point data so that the suction cup of the robot can suck up one material at a time. The suction cup is a structure set at the end of the robot's mechanical arm and is used to suck up the product; the suction cup size includes the length and width of the suction cup.

[0048] In a preferred embodiment, the Open3D image processing algorithm is used in the positioning unit to calculate the point data required for the operation. Specifically, the image acquisition module is first used to collect images of the material to be grasped by the robot, and the image acquisition module transmits the image of the material to be grasped to the visual processing module through the communication module. The positioning unit in the visual processing module calls the Open3D image processing algorithm to convert the image of the material to be grasped into point cloud data, and performs preprocessing such as filtering and feature extraction on the point cloud data to remove noise and extract useful features. Based on the preprocessed point cloud data, a clustering algorithm (such as DBSCAN) is used to perform clustering recognition to obtain a clustering result; the center point of the material to be grasped is extracted from the clustering result as the point data of the target object (also called the grasping site).

[0049] In a preferred embodiment, the image acquisition module integrates the following three different types of cameras: binocular camera, structured light camera and infrared camera. The appropriate camera type can be selected according to the lighting conditions and precision requirements of different industrial environments. For example, an infrared camera is used in a low-light or no-light environment; a binocular camera is selected when medium precision is required in a well-lit environment; a structured light camera is selected in an application scenario where sub-millimeter high-precision measurement is required. In actual use, the hardware model of the actual selected camera is selected in the camera model drop-down box on the camera configuration page, and the IP and port number for communication with the camera are entered and confirmed to save. The entire destacking process package can interact with the camera according to the IP and port number configured by the camera platform and the communication instructions of the corresponding model. At the same time, the camera in the image acquisition module supports users to adjust camera parameters according to actual generation requirements.

[0050] In a preferred embodiment, the image acquisition module is also used to manage three different types of cameras through C++ code factory mode and strategy mode. Combining the factory mode and strategy mode, the image acquisition module can flexibly manage different camera types and can easily expand new camera types while keeping the code clear and maintainable. For example, if a new camera needs to be supported, the developer only needs to add a new camera class and add a new case branch to create an instance in the factory class. For the strategy mode, the corresponding strategy class can be implemented for the new camera and switched to use as needed at runtime.

[0051] It should also be noted that the image acquisition module interacts with the front end through the OATTP framework. Specifically, after the front end receives the "connect camera command" input by the user, it sends the "connect camera command" to the back end; after receiving the "connect camera command", the back end determines whether the IP and port formats are correct. If the IP or port format is incorrect, it returns "3001" to the front end, and the front end pop-up window displays "IP or port error". If the IP and port formats are correct, it is determined whether communication has been established with the camera service. If communication has been established, the O command is sent to open the camera, and "3000" is returned to the front end. The front end pop-up window displays "Operation successful"; if communication has not been established with the camera service, a socket client is established on the back end to establish communication. The specific process of establishing a socket client on the backend is: introduce the header files required by the POSIX sockets library to use functions and data structures related to network communication; use the socket() function to create a new socket; configure the server's IP address and port information in the sockaddr_in structure, and convert the IP address from a string to binary form; initiate a connection request to the server through the connect() function; use the send() and recv() functions to send and receive data between the client and the server; after the communication is completed, use the close() function to close the socket and release resources.

[0052] In a preferred embodiment, Figure 2 As shown, the visual processing module 3 of the system also includes: a visual feedback unit 34, which is used to monitor the real-time grasping path of the robot arm in real time during the robot grasping operation, and adjust the robot's operation strategy according to the real-time grasping path.

[0053] Specifically, the visual feedback unit 34 is also used for: when the real-time grasping path of the robot manipulator is greater than the preset threshold of the robot arm, the grasping path corresponding to the current point data is split into multiple transition paths, and the multiple transition paths are fed back to the robot control system. Because the arm span of the robot manipulator is limited, it means that the working area it can reach is limited. For example, the arm span length of the robot manipulator is 1700mm. When the current position of the robot manipulator (which can be the center position) exceeds 1500mm, or the height difference between the grasping site and the current position of the manipulator is 200mm, walking directly from the current position of the manipulator to the grasping site may cause the manipulator to reach a singular point (that is, some joint angles of the manipulator reach the limit). At this time, the path corresponding to the current grasping site is replanned and split into multiple transition paths using a path planning algorithm, and the multiple transition paths are fed back to the robot control system. The robot control system controls the robot manipulator to move to multiple transition points step by step according to multiple transition paths, and gradually lift to the position of the grasping site.

[0054] In a preferred embodiment, the hand-eye calibration unit interacts with the robot by calling the SDK, and interacts with the image acquisition module through TCP / IP; the hand-eye calibration unit is also used to: send hand-eye calibration instructions to the robot control system, so that the robot control system moves the robot's mechanical arm to a preset position according to the hand-eye calibration instructions; send hand-eye calibration instructions to the image acquisition module, so that the image acquisition module controls the camera to acquire image information of the mechanical arm at a preset position according to the hand-eye calibration instructions; receive image information from the image acquisition module, and calculate the relative position relationship between the camera and the robot based on the image information, so as to realize hand-eye calibration between the camera and the robot.

[0055] Specifically, when performing hand-eye calibration, it is necessary to send a hand-eye calibration instruction to the robot, and the robot control system instructs the robot to move the robotic arm into the workspace; the camera in the image acquisition module collects image information of the robotic arm at two different depths of field in the workspace, ensuring that the collected image information covers enough points and posture information of the X, Y, and Z axes, and the relative position relationship between the camera coordinate system and the robot coordinate system is calculated based on the above information. This relative position relationship includes a rotation matrix and a translation vector. Through the above process, the hand-eye calibration of the eye outside the hand is realized, that is, the camera is fixed at a certain position outside the end of the robot, and the relationship between the camera coordinate system and the robot coordinate system is determined. By using automated hand-eye calibration, manual intervention is reduced, making the calibration process faster and more accurate, and significantly improving work efficiency. By performing multi-point shooting, acquisition and calibration at different depths of field and angles, the operation accuracy of the robot in the entire working area is ensured.

[0056] In a preferred embodiment, Figure 3As shown, the system also includes a user interaction interface 4, which is used to display the operating status, real-time image information, error information and operation guidance information during the robot grasping operation. The current operating status is displayed through an intuitive interface to enhance the user experience and operation transparency. In the depalletizing operation, the user interaction interface 4 serves as an unmanned depalletizing front end, and remote control and monitoring can be achieved by connecting the front end and the system through HTTP. For example, the user can send instructions through the user interaction interface 4 of the front end to query the operating status and real-time image information during the robot grasping operation; if a fault occurs during the robot grasping operation, the error information and operation guidance information can be returned to the user through HTTP. This method makes the depalletizing operation more intelligent and automated, and improves the operation efficiency and accuracy.

[0057] It should be noted that the system also includes a process module, which refers to a software component that controls and manages the process flow. In the process module, thread pool management is used to run in parallel with the robot arm movement. Image processing and path planning can be performed while the robot arm performs other tasks, thereby improving overall work efficiency.

[0058] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0059] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system described in the present application is divided into different functional units or modules to complete all or part of the functions described above.

Claims

1. A visual system used for collaborative robot palletizing, characterized in that: The system comprises: An image acquisition module, a communication module and a visual processing module, wherein the visual processing module is integrated in the robot control system; The image acquisition module is used to call the corresponding camera to acquire images according to the camera type in the camera connection instruction, and three different types of cameras are integrated in the image acquisition module; A communication module, used to establish a communication connection between the image acquisition module and the visual processing module; The visual processing module includes a hand-eye calibration unit, a positioning unit, and an automatic deviation correction unit; A hand-eye calibration unit, used to calibrate the position and posture of the robot according to the image of the robot arm acquired by the image acquisition module; A positioning unit, used to calculate the point data of the target object based on the image of the material to be grasped acquired by the image acquisition module; The automatic deviation correction unit is used to correct the point data so that the robot's suction cup can pick up one material at a time.

2. The visual system for collaborative robot palletizing according to claim 1 is characterized in that: The cameras integrated in the image acquisition module include: binocular camera, structured light camera and infrared camera.

3. The visual system for collaborative robot palletizing according to claim 2 is characterized in that: The image acquisition module is also used to manage three different types of cameras through C++ code factory mode and strategy mode.

4. The visual system for collaborative robot palletizing according to claim 1, characterized in that: The system further comprises: The visual feedback unit is used to monitor the real-time grasping path of the robot arm in real time during the robot grasping operation, and adjust the robot's operation strategy according to the real-time grasping path.

5. The visual system for collaborative robot palletizing according to claim 1, characterized in that: The hand-eye calibration unit is also used for: Sending a hand-eye calibration instruction to the robot control system so that the robot control system moves the robot's mechanical arm to a preset position according to the hand-eye calibration instruction; Sending a hand-eye calibration instruction to the image acquisition module, so that the image acquisition module controls the camera to acquire an image of the robot's mechanical arm at a preset position according to the hand-eye calibration instruction; An image of the robotic arm at a preset position is received, and a relative position relationship between the camera and the robot is calculated based on the image to achieve hand-eye calibration between the camera and the robot.

6. The visual system for collaborative robot palletizing according to claim 5, characterized in that: The hand-eye calibration unit interacts with the robot control system by calling the SDK.

7. The visual system for collaborative robot palletizing according to claim 1, characterized in that: The positioning unit is also used to: convert the image of the target object into point cloud data using the Open3D image processing algorithm; Preprocessing the point cloud data to obtain preprocessed point cloud data; The pre-processed point cloud data is clustered and identified by a clustering algorithm to obtain a clustering result; The center point of the target object is extracted from the clustering result as the point data of the target object.

8. The visual system for collaborative robot palletizing according to claim 4, characterized in that: The visual feedback unit is used to: when the real-time grasping path of the robot arm is greater than a preset threshold of the robot arm, split the grasping path corresponding to the current point data into multiple transition paths, and feed back the multiple transition paths to the robot control system.

9. The visual system for collaborative robot palletizing according to any one of claims 1 to 8, characterized in that: The visual processing module is connected to the image acquisition module via TCP / IP.

10. The visual system for collaborative robot palletizing according to claim 9, characterized in that: The system also includes a user interaction interface, which is used to display the operation status, real-time image information, error information and operation guidance information during the robot's grasping operation.

Images