Camera calibration method, device and equipment in material taking and placing system

Abstract

The invention provides a camera calibration method, device and equipment in a material taking and placing system, and is applied to the technical field of industrial intelligent manufacturing. The method comprises the following steps: carrying out initial calibration on a first camera through a plurality of first images which are acquired by the first camera and assist a calibration object to move; and carrying out initial calibration on the second camera through a plurality of second images which are acquired by the second camera and assist the calibration object to move. A clamping jaw is controlled to grab or place the auxiliary calibration object in the object taking and placing area, and a first camera and a second camera are controlled to collect a first target image and a second target image of the auxiliary calibration object at the same collection moment. And determining a first real scene coordinate of the auxiliary calibration identifier in the component coordinate system through the second target image. Through the first real scene coordinate and the first target image, the camera calibration matrix of the first camera is obtained, and the problem that the robot is difficult to grab and place products due to the fact that the upper camera is fixed can be solved.

Description

Technical Field

[0001] The present application relates to the technical field of industrial intelligent manufacturing, and in particular, to a camera calibration method, device, and equipment in a picking and placing system. Background Art

[0002] With the rapid development of Chinese manufacturing, more and more enterprises begin to use industrial robots to improve production efficiency and product quality. However, traditional robots can only perform set actions, lack flexibility, and cannot adapt to complex and changeable working environments. To solve this problem, engineers have introduced vision sensors (i.e., cameras) to enable robots to observe the surrounding world. By analyzing the images captured by the cameras, robots can identify information such as the position and shape of objects, and adjust their actions accordingly. This approach makes robots seem to have "eyes" and can better adapt to complex and changeable working environments.

[0003] However, the images captured by the cameras are based on their own perspectives (e.g., camera coordinate systems), while robots need to know how to move in their own motion spaces (e.g., gripper coordinate systems), and the conversion between the two is not simple. To enable the vision system and the robot to work together, complex calibration operations are usually required, which are time-consuming and prone to errors. At the same time, if the upper camera is fixed, existing methods may not be able to adapt well to this situation. Summary of the Invention

[0004] The purpose of the present application is to provide a camera calibration method, device, and equipment in a picking and placing system, so as to solve the problem that it is difficult for a robot to grasp and place products when the upper camera is fixed.

[0005] In a first aspect, an embodiment of the present application provides a method for calibrating a camera in a picking and placing system. The picking and placing system includes a first camera, a second camera, and an article picking and placing component. The first camera is fixedly connected to the article picking and placing component. The first camera can move and capture images of the gripper in the article picking and placing component when gripping or placing an article. The second camera is fixed in the article picking and placing area for picking and placing articles. The method includes: performing a first initial calibration on the first camera through a plurality of first images captured by the first camera when the gripper grips an auxiliary calibration object and moves. Performing a second initial calibration on the second camera through a plurality of second images captured by the second camera when the gripper grips an auxiliary calibration object and moves. During the process of controlling the gripper to grip or place the auxiliary calibration object at the article picking and placing area, at the same acquisition moment, respectively control the first camera after the first initial calibration and the second camera after the second initial calibration to capture a first target image and a second target image of the auxiliary calibration object. Through the second target image, determine the first real-scene coordinates of the auxiliary calibration mark on the auxiliary calibration object in the component coordinate system of the article picking and placing component. Through the first real-scene coordinates and the first target image, calibrate the first camera after the first initial calibration to obtain the camera calibration matrix of the first camera.

[0006] A possible implementation manner is that the first target image includes the auxiliary calibration mark on the auxiliary calibration object. The coordinate of the auxiliary calibration mark in the first camera coordinate system is the first image coordinate, and the first camera coordinate system is the coordinate system of the first camera after the first initial calibration. Calibrating the first camera after the first initial calibration through the first real-scene coordinates and the first target image to obtain the camera calibration matrix of the first camera includes: obtaining the first real camera coordinates of the first camera in the component coordinate system of the article picking and placing component. Using the first real camera coordinates, calibrate the first camera after the first initial calibration through the first real-scene coordinates and the first image coordinates to determine the camera calibration matrix of the first camera.

[0007] A possible implementation manner is that the second target image includes the auxiliary calibration mark on the auxiliary calibration object. The coordinate of the auxiliary calibration mark in the second camera coordinate system is the second image coordinate, and the second camera coordinate system is the coordinate system of the second camera after the second initial calibration. Determining the first real-scene coordinates of the auxiliary calibration mark on the auxiliary calibration object in the component coordinate system of the article picking and placing component through the second target image includes: obtaining the second real camera coordinates of the second camera in the component coordinate system of the article picking and placing component. Using the second real camera coordinates, transform the second image coordinates through the nine-point calibration matrix of the second camera obtained by performing the second initial calibration on the second camera to obtain the first real-scene coordinates of the auxiliary calibration mark on the auxiliary calibration object.

[0008] A possible implementation method is to perform a first initial calibration on the first camera by using multiple first images captured by the first camera during the movement of the gripper grasping the auxiliary calibration object. The method includes: during the process of controlling the gripper to grasp the auxiliary calibration object and move in accordance with the preset nine-point calibration sequence at the article picking and placing area, using the first camera to capture the first acquisition image of the gripper. The first acquisition image includes the gripper grasping the auxiliary calibration object, and the coordinate of the auxiliary calibration mark of the auxiliary calibration object in the first acquisition image is the first calibration acquisition coordinate. Obtain the first true coordinate of the auxiliary calibration mark of the auxiliary calibration object in the component coordinate system of the article picking and placing component. Determine the first camera initial calibration matrix of the first camera according to the first calibration acquisition coordinate, the first true coordinate, and the first camera true coordinate of the first camera, so as to complete the first initial calibration of the first camera.

[0009] A possible implementation method is to perform a second initial calibration on the first camera by using multiple second images captured by the second camera during the movement of the gripper grasping the auxiliary calibration object. The method includes: during the process of controlling the gripper to grasp the auxiliary calibration object and move in accordance with the preset nine-point calibration sequence at the article picking and placing area, using the second camera to capture the second acquisition image of the auxiliary calibration object. The second acquisition image includes the auxiliary calibration object, and the coordinate of the auxiliary calibration mark of the auxiliary calibration object in the second acquisition image is the second calibration acquisition coordinate. Obtain the second true coordinate of the auxiliary calibration mark of the auxiliary calibration object in the component coordinate system of the article picking and placing component. Determine the nine-point calibration matrix of the second camera according to the second calibration acquisition coordinate, the second true coordinate, and the second camera true coordinate of the second camera. During the process of controlling the gripper to grasp the auxiliary calibration object and rotate at multiple preset rotation angles at the article picking and placing area, using the second camera to capture the third acquisition image of the auxiliary calibration object. The third acquisition image includes the auxiliary calibration object, and the coordinate of the auxiliary calibration mark of the auxiliary calibration object in the third acquisition image is the third calibration acquisition coordinate. Determine the rotation center of the gripper according to the third calibration acquisition coordinate. Use the rotation center to correct the nine-point calibration matrix to obtain the corrected nine-point calibration matrix, and update the nine-point calibration matrix to the corrected nine-point calibration matrix, so as to complete the second initial calibration of the second camera.

[0010] A possible implementation manner, the method further includes: during the process of controlling the gripper to pick or place the target product at the article pick-and-place area, using the first camera to capture the target product to obtain a product image of the target product. The product image includes the product coordinates of the target product in the coordinate system of the first camera. Using the first camera to capture the product tray to obtain a tray image of the product tray. The tray image includes the tray coordinates of the product tray in the coordinate system of the first camera. According to the product coordinates and the tray coordinates, determine the product rotation angle of the target product. According to the product coordinates and the product rotation angle, determine the product rotation coordinates of the target product. According to the product coordinates, the product rotation coordinates and the tray coordinates, determine the target coordinates of the gripper. Use the target coordinates to control the movement of the gripper so that the gripper picks or places the target product at the target coordinates.

[0011] A possible implementation manner, determining the target coordinates of the gripper according to the product coordinates, the product rotation coordinates and the tray coordinates includes: if the current real coordinates of the first camera are different from the first camera real coordinates of the first camera, determine the correction coordinates of the first camera according to the current real coordinates and the first camera real coordinates. Determine the target coordinates of the gripper according to the product coordinates, the product rotation coordinates, the second camera coordinates of the second camera and the correction coordinates.

[0012] A possible implementation manner, determining the target coordinates of the gripper according to the product coordinates, the product rotation coordinates, the second camera coordinates of the second camera and the correction coordinates includes:

[0013] A1 = S2 - T1 + P1 - Q2 + Q1

[0014] Wherein, A1 is the target coordinates, S2 is the product coordinates, T1 is the product rotation coordinates, P1 is the current real coordinates of the first camera, Q2 is the calibrated coordinates of the first camera, and Q1 is the second camera coordinates.

[0015] In a second aspect, an embodiment of the present application provides a camera calibration device in a pick-and-place system. The pick-and-place system includes a first camera, a second camera and an article pick-and-place component. The first camera is fixedly connected to the article pick-and-place component. The first camera can move and collect images of the gripper in the article pick-and-place component when picking or placing articles. The second camera is fixed in the article pick-and-place area for picking and placing articles. The device includes:

[0016] A calibration module, configured to perform a first initial calibration on the first camera through a plurality of first images collected by the first camera during the movement of the gripper grasping the auxiliary calibration object. Perform a second initial calibration on the second camera through a plurality of second images collected by the second camera during the movement of the gripper grasping the auxiliary calibration object.

[0017] The acquisition module is configured to, during the process of controlling the gripper to grasp or place the auxiliary calibration object at the article picking and placing area, respectively control the first camera after the first initial calibration and the second camera after the second initial calibration to acquire a first target image and a second target image of the auxiliary calibration object at the same acquisition moment.

[0018] The determination module is configured to determine the first real-scene coordinates of the auxiliary calibration mark on the auxiliary calibration object in the component coordinate system of the article picking and placing assembly through the second target image.

[0019] The calibration module is further configured to calibrate the first camera after the first initial calibration through the first real-scene coordinates and the first target image to obtain the camera calibration matrix of the first camera.

[0020] In a third aspect, an embodiment of the present application provides a camera calibration device in a picking and placing system. The camera calibration device has the function of implementing the camera calibration method in the picking and placing system according to the first aspect or any possible implementation manner thereof. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0021] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are run on a computer, the computer can execute the camera calibration method in the picking and placing system according to the first aspect or any possible implementation manner thereof.

[0022] In a fifth aspect, an embodiment of the present application provides a computer program product containing instructions. When the computer program product is run on a computer, the computer can execute the camera calibration method in the picking and placing system according to the first aspect or any possible implementation manner thereof.

[0023] Among them, the technical effects brought by any of the design manners in the second aspect to the fifth aspect can refer to the technical effects brought by different possible implementation manners in the first aspect, which will not be elaborated here. Description of the Drawings

[0024] In order to more clearly illustrate the specific implementation manners of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific implementation manners or the prior art. Obviously, the drawings in the following description are some implementation manners of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0025] Figure 1 It is a structural schematic diagram of a picking and placing system provided by an embodiment of the present application;

[0026] Figure 2 This is a schematic flowchart of a camera calibration method in a pick - and - place system provided by an embodiment of the present application;

[0027] Figure 3 This is a specific example diagram of a camera calibration method in a pick - and - place system provided by an embodiment of the present application;

[0028] Figure 4 This is another specific example diagram of a camera calibration method in a pick - and - place system provided by an embodiment of the present application;

[0029] Figure 5 This is a schematic structural diagram of a camera calibration device in a pick - and - place system provided by an embodiment of the present application;

[0030] Figure 6 This is a schematic structural diagram of a camera calibration system in a pick - and - place system provided by an embodiment of the present application. Detailed implementation manners

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. Usually, the components of the embodiments of the present application described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations.

[0032] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application claimed, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0033] The robot vision system actively collects image information of the robot's working environment and processes this image information using intelligent vision technology to obtain the data required for robot operation. This enables the robot to execute tasks more flexibly and precisely, greatly expanding its application scenarios. In the field of intelligent manufacturing, the pick - and - place guiding technology is usually combined with an automated production line to achieve a highly automated production process.

[0034] However, although significant progress has been made in guiding pick-and-place technology, there are still some deficiencies in practical applications. For example, the conversion from image coordinates to actual coordinates has not been achieved. For instance, the guiding pick-and-place system can only judge the difference in image coordinates between the current photo and the first photo taken, and cannot provide an effective guiding function. It can only be used for simple tasks such as product presence / absence detection and character recognition. The traditional correlation calibration method is only applicable when the upper camera is fixed on the robot arm or the module. When the upper camera is stationary, the traditional correlation calibration method is not applicable.

[0035] On the one hand, an embodiment of the present application provides a pick-and-place system. As Figure 1 shown, the pick-and-place system 100 may include a first camera 101, a second camera 102, and an article pick-and-place component 103.

[0036] Among them, the first camera 101 is fixed at a preset spatial position. For example, the first camera is fixed at any position of the article pick-and-place component. The first camera 101 can move horizontally along with the article pick-and-place component 103 to complete the shooting during the process of the article pick-and-place component 103 grasping or placing products on multiple product trays. And the distance between the first camera 101 and the article pick-and-place component 103 is equal to the preset distance. The first camera 101 can be used to provide a close-up view when the article pick-and-place component approaches the target object, assisting the article pick-and-place component to accurately locate the target object. The first camera 101 is also used to monitor the actions of the article pick-and-place component in real time to ensure the safety and accuracy of the article pick-and-place component during the process of grasping the target object. The first camera 101 can also be used to provide a feedback mechanism by monitoring the actions of the article pick-and-place component, assisting in adjusting the posture of the article pick-and-place component to handle target objects of different shapes and sizes.

[0037] The second camera 102 is fixed on the plane where the target object to be grasped is located, such as a workbench surface or a conveyor belt, etc. The second camera 102 can be used to assist the first camera 101 in calibration.

[0038] The article pick-and-place component 103 may include a driving device and a gripper. The gripper is controlled by the driving device and can move flexibly in three-dimensional space to perform precise grasping and placing actions. The driving device can be, for example, a robotic arm. The article pick-and-place component 103 can be designed into different forms according to application requirements, such as a parallel article pick-and-place component, a vacuum chuck, etc., and can be applicable to target objects of various materials and shapes. The article pick-and-place component 103 can also be equipped with a torque sensor and a pressure sensor, etc. By monitoring the grasping force of the article pick-and-place component 103 in real time, it is possible to prevent damage to the target object or its own components.

[0039] It should be noted that the above Figure 1The schematic picking and placing system 100 is only an example for illustrating the application scenario of the solution of the present application, rather than a limitation on the application scenario of the solution of the present application.

[0040] On the one hand, an embodiment of the present application provides a method for calibrating a camera in a picking and placing system, and this method can be executed by Figure 1 the shown picking and placing system 100. As Figure 2 shown, this method includes:

[0041] S201, perform a first initial calibration on the first camera by using a plurality of first images collected by the first camera during the movement of the gripper grasping the auxiliary calibration object.

[0042] Exemplarily, first configure and calibrate the first camera and the second camera. For example, set appropriate parameters such as focal length, exposure time, and gain so that the images collected by the first camera and the second camera are clear, have appropriate contrast, and are not distorted.

[0043] After configuring and calibrating the first camera, during the process of controlling the gripper to grasp the auxiliary calibration object and move in the item picking and placing area according to the preset nine-point calibration sequence, use the first camera to collect the first acquisition images of the gripper. For example, control the gripper to grasp the auxiliary calibration object and move in the order of Figure 3 marking points 1 to 9 in

[0044] to obtain the images of the auxiliary calibration object under each marking point.

[0045] Among them, the first acquisition image includes the gripper grasping the auxiliary calibration object, and the coordinates of the auxiliary calibration mark of the auxiliary calibration object in the first acquisition image are the first calibration acquisition coordinates.

[0046] It should be noted that the auxiliary calibration mark can also be any reference point on the auxiliary calibration object, such as the reference point or edge point of the auxiliary calibration object. The first calibration acquisition coordinates can also be any reference point on the gripper, such as the center point of the gripper.

[0047] Obtain the first true coordinates of the auxiliary calibration mark of the auxiliary calibration object in the component coordinate system of the item picking and placing component.

[0048] According to the first calibration acquisition coordinates, the first true coordinates, and the first camera true coordinates of the first camera, determine the first camera initial calibration matrix of the first camera to complete the first initial calibration of the first camera.

[0049] During the process of controlling the gripper to grasp and move the auxiliary calibration object in the item picking and placing area according to the preset nine-point calibration sequence, use the second camera to collect the second acquisition image of the auxiliary calibration object. For example, according to Figure 3 the order of the marked points 1 to 9 in

[0050] control the gripper to grasp and move the target object. Among them, the second acquisition image includes the auxiliary calibration object, and the coordinates of the auxiliary calibration identifier of the auxiliary calibration object in the second acquisition image are the second calibration acquisition coordinates.

[0051] Obtain the second true coordinates of the auxiliary calibration identifier of the auxiliary calibration object in the component coordinate system of the item picking and placing component.

[0052] According to the second calibration acquisition coordinates, the second true coordinates, and the second true coordinates of the second camera, determine the nine-point calibration matrix of the second camera.

[0053] Furthermore, during the process of controlling the gripper to grasp and rotate the auxiliary calibration object at a preset multiple of rotation angles in the item picking and placing area, use the second camera to collect the third acquisition image of the auxiliary calibration object.

[0054] For example, control the gripper to grasp and rotate the target object according to the multiple of rotation angles (such as -45°, -30°, 0°, 30°, 45°, etc.) as shown in Figure 4 . Among them, the marked point 10 in Figure 4 is the same as any marked point in Figure 3 .

[0055] Among them, the third acquisition image includes the auxiliary calibration object, and the coordinates of the auxiliary calibration identifier of the auxiliary calibration object in the third acquisition image are the third calibration acquisition coordinates.

[0056] Determine the rotation center of the gripper according to the third calibration acquisition coordinates.

[0057] Finally, use the rotation center to correct the nine-point calibration matrix to obtain the corrected nine-point calibration matrix, and update the nine-point calibration matrix to the corrected nine-point calibration matrix to complete the second initial calibration of the second camera.

[0058] S203. During the process of controlling the gripper to grasp or place the auxiliary calibration object in the item picking and placing area, at the same acquisition moment, respectively control the first camera after the first initial calibration and the second camera after the second initial calibration to collect the first target image and the second target image of the auxiliary calibration object.

[0059] Exemplarily, at a specific moment when the gripper performs a grasping or placing action, a synchronization signal is sent through the control system to trigger image acquisition by the first camera after the first initial calibration and the second camera after the second initial calibration at the same acquisition moment, obtaining a first target image and a second target image of the auxiliary calibration object.

[0060] This process can, for example, trigger the first camera after the first initial calibration and the second camera after the second initial calibration to perform image acquisition at the same acquisition moment through a hardware synchronization line (such as a GPIO signal) or a software synchronization command (such as sending a unified timestamp).

[0061] S203. Determine the first true scene coordinates of the auxiliary calibration mark on the auxiliary calibration object in the component coordinate system of the article picking and placing assembly through the second target image.

[0062] Among them, the second target image includes the auxiliary calibration mark on the auxiliary calibration object. The coordinates of the auxiliary calibration mark in the second camera coordinate system are the second image coordinates, and the second camera coordinate system is the coordinate system of the second camera after the second initial calibration.

[0063] A possible implementation method is to obtain the true coordinates of the second camera in the component coordinate system of the article picking and placing assembly.

[0064] Using the true coordinates of the second camera and the nine-point calibration matrix of the second camera obtained through the second initial calibration of the second camera, convert the second image coordinates to obtain the first true scene coordinates of the auxiliary calibration mark on the auxiliary calibration object.

[0065] S204. Calibrate the first camera after the first initial calibration through the first true scene coordinates and the first target image to obtain the camera calibration matrix of the first camera.

[0066] Among them, the first target image includes the auxiliary calibration mark on the auxiliary calibration object. The coordinates of the auxiliary calibration mark in the first camera coordinate system are the first image coordinates, and the first camera coordinate system is the coordinate system of the first camera after the first initial calibration.

[0067] A possible implementation method is to obtain the true coordinates of the first camera in the component coordinate system of the article picking and placing assembly.

[0068] Using the true coordinates of the first camera, the first true scene coordinates, and the first image coordinates, calibrate the first camera after the first initial calibration to determine the camera calibration matrix of the first camera.

[0069] The calibration method of the camera provided by the embodiments of the present application controls the gripper to grab the target object for movement by using the order of nine-point calibration, and respectively collects and shoots the target object by using the first camera and the second camera. When the first camera is fixed, the calibration matrix of the first camera can be determined by using the collected images, improving the calibration of the first camera.

[0070] Further, when the gripper actually grabs and places the target product, only the first camera and the gripper are included in the loading and unloading system at this time, and the second camera is only used to assist in determining the calibration matrix of the first camera.

[0071] The first camera is used to shoot the target product to obtain the product image of the target product. The product image includes the product coordinates of the target product in the coordinate system of the first camera. The product image includes the product coordinates of the target product in the coordinate system of the first camera.

[0072] The first camera is used to shoot the product tray to obtain the tray image of the product tray. The tray image includes the tray coordinates of the product tray in the coordinate system of the first camera. According to the product coordinates and the tray coordinates, the product rotation angle of the target product is determined. According to the product coordinates and the product rotation angle, the product rotation coordinates of the target product are determined. According to the product coordinates, the product rotation coordinates and the tray coordinates, the discharging coordinates of the gripper are determined.

[0073] After determining the discharging coordinates, the gripper is controlled to move by using the discharging coordinates.

[0074] Exemplarily, the first camera is used to shoot the target product to obtain the image of the product (i.e., the product image), which contains the product coordinates of the target product in the coordinate system of the first camera.

[0075] The first camera is used to shoot the tray where the product is located to obtain the tray image, which contains the tray coordinates of the tray in the coordinate system of the first camera. According to the product coordinates and the tray coordinates, the product rotation angle of the target product relative to the tray is determined through geometric analysis or feature matching algorithm.

[0076] Combining the product coordinates and the product rotation angle, the rotated coordinates of the target product in the gripper coordinate system, that is, the product rotation coordinates, are calculated.

[0077] Further, in order to ensure that the gripper can accurately place the target product at the specified position, the system needs to calculate the final discharging coordinates. By using the product coordinates, the product rotation coordinates and the tray coordinates, the precise coordinates of the gripper when performing the discharging action, that is, the discharging coordinates, are calculated through the coordinate transformation formula. The gripper accurately places the target product at the predetermined position according to the calculated discharging coordinates, completing the entire loading and unloading process.

[0078] Further, if the current real coordinates of the first camera are different from the real coordinates of the first camera, the corrected coordinates of the first camera are determined based on the current real coordinates and the real coordinates of the first camera. The target coordinates of the gripper are determined based on the product coordinates, the product rotation coordinates, the coordinates of the second camera of the second camera, and the corrected coordinates.

[0079] For example, the target coordinates of the gripper can be determined by the following equation.

[0080] A1 = S2 - T1 + P1 - Q2 + Q1

[0081] Where A1 is the target coordinates, S2 is the product coordinates, T1 is the product rotation coordinates, P1 is the current real coordinates of the first camera, Q2 is the calibrated coordinates of the first camera, and Q1 is the coordinates of the second camera.

[0082] For example, when controlling the gripper to pick up the target product at the item picking and placing area, the first camera has been calibrated at this time. First, use the first camera to take a picture of the target product in the tray to obtain the first image coordinates of the product. After converting the first image coordinates of the target product through the calibration matrix of the first camera, the product coordinates K1(x1, y1, r1) of the product are obtained. Then the gripper picks up the target product in the tray. At this time, the picking coordinates of the gripper are: K1 + the first camera photographing position coordinates when the first camera takes a picture of the product in the tray - the first camera photographing position when the first camera is calibrated.

[0083] When controlling the gripper to place the target product at the item picking and placing area, if it is a scenario with low requirements for placing accuracy, only the first camera is used to assist the gripper in placing the material at this time. After the first camera takes a picture of the tray, the gripper (the target product is already on the gripper at this time) places the target product on the tray.

[0084] When controlling the gripper to place the target product at the item picking and placing area, if it is a scenario with high requirements for placing accuracy, the second camera is needed to assist in placing the target product. At this time, the photographing position coordinates of the second camera are Q1, and the calibrated photographing position coordinates of the first camera are Q2. The gripper (the target product is already on the gripper at this time) moves above the second camera, and the second camera takes a picture to record the product image coordinates D1(x1, y1, r1) of the target product obtained by the second camera taking a picture. Then the first camera takes a picture of the tray to obtain the image coordinates U1(x2, y2, r2) of the tray (product placing position), and records the first camera photographing position coordinates P1(x3, y3, r3) at this time.

[0085] Furthermore, according to the camera calibration matrix of D1 and the second camera, the true coordinates S1(x11, y11, r11) corresponding to D1 are determined. According to D2 and the camera calibration matrix of the first camera, the true coordinates S2(x22, y22, r22) corresponding to D2 are determined. The rotation angle of the target product is determined as R = r22 - r11, and according to the rotation angle, the rotation coordinates of the target product are determined as the coordinates of S1 after rotating by the angle R around Q1. Finally, the product coordinates of the target product are determined as A1 = S2 - T1 + P1 - Q2 + Q1. Furthermore, the gripper is controlled to discharge materials at the coordinates of A1.

[0086] The above mainly introduces the solution provided in the embodiments of the present application from the perspective of the working principle of the camera calibration device in the loading and unloading system. It can be understood that in order to implement the above functions, the camera calibration device in the loading and unloading system includes the corresponding hardware structure and / or software module for executing each function. Those skilled in the art should easily realize that, in combination with the algorithm steps of the examples described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0087] The embodiments of the present application can divide the functional modules of the camera calibration device in the loading and unloading system according to the above method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processor. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0088] It should be noted that the division of modules in the embodiments of the present application is illustrative, only a logical function division, and there may be other division methods in actual implementation. In the case of dividing each functional module corresponding to each function, Figure 5 shows a possible composition schematic diagram of the camera calibration device in the loading and unloading system involved in the above and the embodiments. As Figure 5 shown, the camera calibration device 500 in the loading and unloading system may include: a calibration module 501, a collection and determination module 502, and a determination module 503.

[0089] Among them, the collection module 501 is used to support the camera calibration device 500 in the loading and unloading system provided in the embodiments of the present application to execute S201, S202, and S205 in the camera calibration method in the loading and unloading system as Figure 2 shown.

[0090] The acquisition module 502 is used to support the camera calibration device 500 in the pick-and-place system provided by the embodiments of the present application to execute S203 in the camera calibration method in the pick-and-place system as Figure 2 shown in

[0091] The determination module 503 is used to support the camera calibration device 500 in the pick-and-place system provided by the embodiments of the present application to execute S204 in the camera calibration method in the pick-and-place system as Figure 2 shown in

[0092] In a possible implementation, the first target image includes an auxiliary calibration mark on the auxiliary calibration object, and the coordinates of the auxiliary calibration mark in the first camera coordinate system are the first image coordinates. The first camera coordinate system is the coordinate system of the first camera after the first initial calibration. The device is specifically configured to obtain the first real camera coordinate of the first camera in the component coordinate system of the article pick-and-place component. Using the first real camera coordinate, through the first real scene coordinate and the first image coordinate, calibrate the first camera after the first initial calibration, and determine the camera calibration matrix of the first camera.

[0093] In a possible implementation, the second target image includes an auxiliary calibration mark on the auxiliary calibration object, and the coordinates of the auxiliary calibration mark in the second camera coordinate system are the second image coordinates. The second camera coordinate system is the coordinate system of the second camera after the second initial calibration. The device is specifically configured to obtain the second real camera coordinate of the second camera in the component coordinate system of the article pick-and-place component. Using the second real camera coordinate, through the nine-point calibration matrix of the second camera obtained by performing the second initial calibration on the second camera, transform the second image coordinates to obtain the first real scene coordinate of the auxiliary calibration mark on the auxiliary calibration object.

[0094] In a possible implementation, the device is specifically configured to, during the process of controlling the gripper to grasp and move the auxiliary calibration object in accordance with the preset nine-point calibration sequence at the article pick-and-place area, use the first camera to collect the first acquisition image of the gripper. The first acquisition image includes the gripper grasping the auxiliary calibration object, and the coordinates of the auxiliary calibration mark of the auxiliary calibration object in the first acquisition image are the first calibration acquisition coordinates. Obtain the first real coordinate of the auxiliary calibration mark of the auxiliary calibration object in the component coordinate system of the article pick-and-place component. According to the first calibration acquisition coordinates, the first real coordinate, and the first real camera coordinate of the first camera, determine the first camera initial calibration matrix of the first camera to complete the first initial calibration of the first camera.

[0095] A possible implementation mode, the device is specifically used for controlling the gripper to grasp and move the auxiliary calibration object in the order of preset nine-point calibration at the article picking and placing area, and collecting the second acquisition image of the auxiliary calibration object by using the second camera. The second acquisition image includes the auxiliary calibration object, and the coordinates of the auxiliary calibration identifier of the auxiliary calibration object in the first acquisition image are the second calibration acquisition coordinates. Obtain the second true coordinates of the auxiliary calibration identifier of the auxiliary calibration object in the component coordinate system of the article picking and placing component. Determine the nine-point calibration matrix of the second camera according to the second calibration acquisition coordinates, the second true coordinates and the second true coordinates of the second camera. When controlling the gripper to grasp and rotate the auxiliary calibration object at multiple preset rotation angles at the article picking and placing area, collect the third acquisition image of the auxiliary calibration object by using the second camera. The third acquisition image includes the auxiliary calibration object, and the coordinates of the auxiliary calibration identifier of the auxiliary calibration object in the third acquisition image are the third calibration acquisition coordinates. Determine the rotation center of the gripper according to the third calibration acquisition coordinates. Use the rotation center to correct the nine-point calibration matrix to obtain the corrected nine-point calibration matrix, and update the nine-point calibration matrix to the corrected nine-point calibration matrix to complete the second initial calibration of the second camera.

[0096] A possible implementation mode, the device is specifically used for controlling the gripper to pick or place the target product at the article picking and placing area, and using the first camera to shoot the target product to obtain the product image of the target product. The product image includes the product coordinates of the target product in the first camera coordinate system. Use the first camera to shoot the product tray to obtain the tray image of the product tray. The tray image includes the tray coordinates of the product tray in the first camera coordinate system. Determine the product rotation angle of the target product according to the product coordinates and the tray coordinates. Determine the product rotation coordinates of the target product according to the product coordinates and the product rotation angle. Determine the target coordinates of the gripper according to the product coordinates, the product rotation coordinates and the tray coordinates. Control the movement of the gripper by using the target coordinates so that the gripper picks or places the target product at the target coordinates.

[0097] A possible implementation mode, the device is specifically used for if the current true coordinates of the first camera are different from the first camera true coordinates of the first camera, determining the corrected coordinates of the first camera according to the current true coordinates and the first camera true coordinates. Determine the target coordinates of the gripper according to the product coordinates, the product rotation coordinates, the second camera coordinates of the second camera and the corrected coordinates.

[0098] A possible implementation mode, the device is specifically used for

[0099] A1 = S2 - T1 + P1 - Q2 + Q1

[0100] Wherein, A1 is the target coordinate, S2 is the product coordinate, T1 is the product rotation coordinate, P1 is the current actual coordinate of the first camera, Q2 is the calibration coordinate of the first camera, and Q1 is the coordinate of the second camera.

[0101] It should be noted that all relevant contents of each step involved in the above method embodiments can be cited in the function descriptions of the corresponding functional modules, and will not be elaborated here.

[0102] The camera calibration device 500 in the pick-and-place system provided by the embodiments of the present application is used to execute the above Figure 2 shown camera calibration method, so the same effect as the above camera calibration method can be achieved.

[0103] The embodiments of the present application also provide a camera calibration device in a pick-and-place system. The camera calibration device in the pick-and-place system can execute the camera calibration method and related steps in the above method embodiments.

[0104] The embodiments of the present application also provide a computer-readable storage medium, on which instructions are stored. When the instructions are executed, the camera calibration method and related steps in the above method embodiments of the pick-and-place system are executed.

[0105] The embodiments of the present application also provide a computer program product. When the computer program product runs on a computer, the computer is caused to execute the camera calibration method and related steps in the above method embodiments.

[0106] In some embodiments, the method shown in the present application can be implemented as computer program instructions encoded in a computer-readable storage medium in a machine-readable format or encoded in other non-transitory media or articles.

[0107] The embodiments of the present application also provide a camera calibration system 600 in a pick-and-place system, as Figure 6 shown. The camera calibration system 600 in the pick-and-place system includes at least one processor 601 and at least one interface circuit 602.

[0108] As an example, when the camera calibration system 600 in the pick-and-place system includes one processor and one interface circuit, then the one processor can be Figure 6 the processor 601 shown in the solid line box (or the processor 601 shown in the dashed line box), and the one interface circuit can be Figure 6 the interface circuit 602 shown in the solid line box (or the interface circuit 602 shown in the dashed line box). When the camera calibration system 200 includes two processors and two interface circuits, then the two processors include Figure 6The processor 601 shown by the solid line box and the processor 601 shown by the dashed line box. The two interface circuits include Figure 6 the interface circuit 602 shown by the solid line box and the interface circuit 602 shown by the dashed line box. There is no limitation on this.

[0109] The processor 601 and the interface circuit 602 can be interconnected through lines. For example, the interface circuit 602 can be used to receive signals. Also, for example, the interface circuit 602 can be used to send signals to other devices (such as the processor 601). By way of example, the interface circuit 602 can read the computer instructions stored in the memory and send the computer instructions to the processor 601. The processor 601 executes the instructions and, in combination with the input / output device, implements each step in the above embodiments, such as implementing Figures 2 to 4 each step executed in any of the method embodiments shown. Of course, the camera calibration system 600 in the picking and placing system may further include other discrete devices, and the embodiments of the present application do not make specific limitations on this.

[0110] Through the description of the above embodiments, those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0111] In several embodiments provided in the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0112] The units described as separate components may or may not be physically separated. The components displayed as units may be one physical unit or multiple physical units, that is, they can be located in one place, or they can be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0113] In addition, in each embodiment of the present application, the functional units may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0114] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present application, in essence, or the part that makes a contribution, or all or part of the technical solution, may be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions for causing a camera calibration device (which may be a single-chip microcomputer, a chip, etc.) or a processor in a loading and unloading system to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc that can store program codes.

[0115] The above content is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A camera calibration method in a material handling system, characterized in that: The material picking and placing system comprises a first camera, a second camera and an item picking and placing assembly; the first camera is fixedly connected to the item picking and placing assembly, the first camera can move and collect images of the gripper in the item picking and placing assembly grabbing or placing items, and the second camera is fixed in an item picking and placing area for picking and placing items. The method comprises: Performing a first initial calibration on the first camera through a plurality of first images captured by the first camera and in which the clamp grasps the auxiliary calibration object in motion; Performing a second initial calibration on the second camera through a plurality of second images captured by the second camera and in which the gripper grasps the auxiliary calibration object in motion; In the process of controlling the gripper to grasp or place the auxiliary calibration object at the object placement area, the first camera after the first initial calibration and the second camera after the second initial calibration are respectively controlled to acquire the first target image and the second target image of the auxiliary calibration object at the same acquisition time; Determining, by means of the second target image, a first real scene coordinate of the auxiliary calibration mark on the auxiliary calibration object in a component coordinate system of the item picking and placing component; The first camera after the first initial calibration is calibrated using the first real scene coordinates and the first target image to obtain a camera calibration matrix of the first camera.

2. The method according to claim 1, characterized in that The first target image includes an auxiliary calibration mark on the auxiliary calibration object, the coordinates of the auxiliary calibration mark in the first camera coordinate system are first image coordinates, and the first camera coordinate system is a coordinate system of the first camera after the first initial calibration; The step of calibrating the first camera after the first initial calibration by using the first real scene coordinates and the first target image to obtain a camera calibration matrix of the first camera includes: Obtaining the real coordinates of the first camera of the first camera in the component coordinate system of the item picking and placing component; The first camera after the first initial calibration is calibrated using the first camera real coordinates, the first real scene coordinates and the first image coordinates to determine a camera calibration matrix of the first camera.

3. The method according to claim 1, characterized in that The second target image includes the auxiliary calibration mark on the auxiliary calibration object, the coordinates of the auxiliary calibration mark in the second camera coordinate system are second image coordinates, and the second camera coordinate system is the coordinate system of the second camera after the second initial calibration is performed; Determining, by using the second target image, a first real scene coordinate of the auxiliary calibration mark on the auxiliary calibration object in the component coordinate system of the item picking and placing component includes: Obtain the real coordinates of the second camera of the second camera in the component coordinate system of the item picking and placing component; The second image coordinates are transformed by using the real coordinates of the second camera and the nine-point calibration matrix of the second camera obtained by performing a second initial calibration on the second camera to obtain the auxiliary calibration mark on the auxiliary calibration object at the first real scene coordinates.

4. The method according to claim 1, characterized in that: The first camera is captured by the first camera and multiple first images of the auxiliary calibration object being grasped by the gripper are used to perform a first initial calibration on the first camera, including: In the process of controlling the movement of the auxiliary calibration object by the clamping claw according to the preset nine-point calibration sequence at the object picking and placing area, the first camera is used to capture a first acquisition image of the clamping claw; the first acquisition image includes the auxiliary calibration object being captured by the clamping claw, and the coordinates of the auxiliary calibration mark of the auxiliary calibration object in the first acquisition image are the first calibration acquisition coordinates; Acquire a first real coordinate of the auxiliary calibration mark of the auxiliary calibration object in the component coordinate system of the item picking and placing component; A first camera initial calibration matrix of the first camera is determined according to the first calibration acquisition coordinates, the first real coordinates and the first camera real coordinates of the first camera to complete the first initial calibration of the first camera.

5. The method according to claim 1, characterized in that The second camera captures a plurality of second images of the auxiliary calibration object being grasped by the gripper while it is moving, and performs a second initial calibration on the first camera, comprising: In the process of controlling the gripper to grasp the auxiliary calibration object and move it in the order of the preset nine-point calibration at the item pick-up and placement area, the second camera is used to capture a second captured image of the auxiliary calibration object; the second captured image includes the auxiliary calibration object, and the coordinates of the auxiliary calibration mark of the auxiliary calibration object in the second captured image are the second calibration captured coordinates; Acquire a second real coordinate of the auxiliary calibration mark of the auxiliary calibration object in the component coordinate system of the item picking and placing component; Determine a nine-point calibration matrix of the second camera according to the second calibration acquisition coordinates, the second real coordinates, and the second camera real coordinates of the second camera; In the process of controlling the gripper to grasp the auxiliary calibration object and rotate it according to a plurality of preset rotation angles at the item placement area, using the second camera to capture a third captured image of the auxiliary calibration object; the third captured image includes the auxiliary calibration object, and the coordinates of the auxiliary calibration mark of the auxiliary calibration object in the third captured image are third calibration captured coordinates; Determining the rotation center of the gripper according to the third calibration acquisition coordinates; The nine-point calibration matrix is ​​corrected using the rotation center to obtain the corrected nine-point calibration matrix, and the nine-point calibration matrix is ​​updated to the corrected nine-point calibration matrix to complete the second initial calibration of the second camera.

6. The method according to claim 1, characterized in that The method further comprises: In the process of controlling the gripper to pick up or place a target product at the item pick-up and placement area, the target product is photographed by the first camera to obtain a product image of the target product; the product image includes the product coordinates of the target product in the first camera coordinate system; Using the first camera to photograph the product tray, to obtain a tray image of the product tray; the tray image includes the tray coordinates of the product tray in the first camera coordinate system; Determining a product rotation angle of the target product according to the product coordinates and the tray coordinates; Determining the product rotation coordinates of the target product according to the product coordinates and the product rotation angle; Determining the target coordinates of the gripper according to the product coordinates, the product rotation coordinates and the tray coordinates; The target coordinates are used to control the movement of the gripper so that the gripper picks up or places the target product at the target coordinates.

7. The method according to claim 6, characterized in that Determining the target coordinates of the gripper according to the product coordinates, the product rotation coordinates and the tray coordinates comprises: If the current real coordinates of the first camera are different from the first camera real coordinates of the first camera, determining the corrected coordinates of the first camera according to the current real coordinates and the first camera real coordinates; The target coordinates of the gripper are determined according to the product coordinates, the product rotation coordinates, the second camera coordinates of the second camera and the corrected coordinates.

8. The method according to claim 7, characterized in that The step of determining the target coordinates of the gripper according to the product coordinates, the product rotation coordinates, the second camera coordinates of the second camera and the corrected coordinates includes: A1=S2-T1+P1-Q2+Q1 Among them, A1 is the target coordinate, S2 is the product coordinate, T1 is the product rotation coordinate, P1 is the current real coordinate of the first camera, Q2 is the calibrated coordinate of the first camera, and Q1 is the coordinate of the second camera.

9. A camera calibration device in a material handling system, characterized in that: The material picking and placing system comprises a first camera, a second camera and an article picking and placing assembly; the first camera is fixedly connected to the article picking and placing assembly, the first camera can move and collect images of the gripper in the article picking and placing assembly grabbing or placing an article, the second camera is fixed in an article picking and placing area for picking and placing an article, and the device comprises: A calibration module, configured to perform a first initial calibration on the first camera through a plurality of first images captured by the first camera and captured by the gripper while the auxiliary calibration object is moving; and to perform a second initial calibration on the second camera through a plurality of second images captured by the second camera and captured by the gripper while the auxiliary calibration object is moving; A collection module, used for controlling the first camera after the first initial calibration and the second camera after the second initial calibration to collect the first target image and the second target image of the auxiliary calibration object at the same collection time in the process of controlling the gripper to grasp or place the auxiliary calibration object at the object placement area; a determination module, configured to determine, through the second target image, a first real scene coordinate of the auxiliary calibration mark on the auxiliary calibration object in the component coordinate system of the item picking and placing component; The calibration module is further used to calibrate the first camera after the first initial calibration through the first real scene coordinates and the first target image to obtain a camera calibration matrix of the first camera.

10. A camera calibration device in a material handling system, characterized in that: The camera calibration device includes a processor and a memory, the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the steps of the camera calibration method in the material loading and unloading system according to any one of claims 1 to 8.

Images