Camera calibration method and device, computer equipment and storage medium

Abstract

The invention relates to a camera calibration method and device, computer equipment, a storage medium and a computer program product. The method comprises the steps that under the condition that the center point of a target object and the plane center of an image plane of a camera meet the coincidence condition, an initial position parameter of a mechanical arm where the camera is located at an initial working point is obtained, under the condition that the target object rotates for multiple times, image recognition is conducted on the rotated target object, and the image recognition result is obtained. According to the initial position parameters, the rotating angles of all the rotated target objects are obtained, the positions of the mechanical arms corresponding to all the rotated target objects are positioned based on the initial position parameters and all the rotating angles, and the running position parameters of the mechanical arms are obtained; according to the method, the operation working point corresponding to the operation position parameter, the position of the camera, the angle deviation formed among the initial working point and the pixel coordinate deviation are obtained, the camera is calibrated according to the multiple groups of pixel coordinate deviations and angle deviations, errors generated in the calibration process are reduced, and the camera calibration precision is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a camera calibration method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] The vision system where the vision camera is located is often used to assist the robotic arm in positioning, processing, and detection operations. Before using the vision system to assist the robotic arm in positioning, processing, and detection operations, the vision camera needs to be calibrated.

[0003] The traditional calibration method involves rotating the robotic arm and using the visual camera on the robotic arm to collect images of multiple rotation angles of the robotic arm. The calibration parameters are calculated based on the differences between the target points contained in the multiple images. When the camera is calibrated based on the calibration parameters, the camera calibration accuracy is still low. Summary of the invention

[0004] Based on this, it is necessary to provide a camera calibration method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the camera calibration accuracy in response to the above technical problems.

[0005] In a first aspect, the present application provides a camera calibration method. The method comprises:

[0006] When the center point of the target object and the center of the image plane of the camera meet the coincidence condition, the initial position parameters of the robotic arm where the camera is located at the initial working point are obtained; when the target object rotates multiple times, image recognition is performed on each rotated target object to obtain the rotation angle of each rotated target object; based on the initial position parameters and the rotation angles, the robotic arms corresponding to each rotated target object are positioned to obtain the operating position parameters of each robotic arm; for each operating position parameter, the operating working point corresponding to the operating position parameter, the position of the camera corresponding to the operating position parameter, the angle deviation between the initial working point, and the pixel coordinate deviation between the center point of the target object matched by the operating position parameter and the center of the plane are obtained; based on the pixel coordinate deviations and the angle deviations, the camera is calibrated.

[0007] In one embodiment, performing image recognition on each rotated target to obtain the rotation angle of each rotated target includes: determining the rotation center point of each rotated target; and performing image recognition on each rotated target based on each rotation center point to obtain the rotation angle of each rotated target.

[0008] In one embodiment, based on each of the rotation center points, image recognition is performed on each of the rotated targets to obtain the rotation angle of each of the rotated targets, including: contour extraction of the target to determine the initial contour of the target before rotation; contour extraction of each of the rotated targets to obtain the target contour of each of the rotated targets; for each of the rotated targets, contour matching the initial contour with the corresponding target contour feature to obtain the contour difference between the target contour and the initial contour; based on the contour difference corresponding to each of the rotated targets and each of the rotation center points, the rotation angle of each of the rotated targets is determined.

[0009] In one embodiment, the camera is calibrated based on each of the pixel coordinate deviations and each of the angle deviations, including: obtaining an affine transformation matrix between a pixel coordinate system and a robotic arm coordinate system; based on the affine transformation matrix, mapping each of the pixel coordinate deviations respectively to obtain a machine coordinate deviation of each of the pixel coordinate deviations in the robotic arm coordinate system; based on each of the machine coordinate deviations and each of the angle deviations, determining a physical compensation vector of the camera; and updating the calibration data of the camera based on the physical compensation vector.

[0010] In one of the embodiments, based on the affine transformation matrix, each of the pixel coordinate deviations is mapped separately to obtain the machine coordinate deviation of each of the pixel coordinate deviations in the robotic arm coordinate system, including: obtaining a pixel deviation threshold configured for the target object; when the number of pixel coordinate deviations greater than the pixel deviation threshold satisfies a preset value, based on the affine transformation matrix, each of the pixel coordinate deviations is mapped separately to obtain the machine coordinate deviation of each of the pixel coordinate deviations in the robotic arm coordinate system.

[0011] In one embodiment, the position of the camera includes the camera mechanical coordinates of the camera in the robotic arm coordinate system; the obtaining of the operating working point corresponding to the operating position parameter, the position of the camera corresponding to the operating position parameter, and the angular deviation between the initial working point includes: determining the spatial coordinate correspondence between the camera and the robotic arm; obtaining the operating mechanical coordinates of the operating working point corresponding to the operating position parameter in the robotic arm coordinate system, and the initial mechanical coordinates of the initial working point in the robotic arm coordinate system; determining the camera mechanical coordinates based on the initial mechanical coordinates and the spatial coordinate correspondence; determining the operating working point corresponding to the operating position parameter, the position of the camera corresponding to the operating position parameter, and the angular deviation between the initial working point based on the operating mechanical coordinates, the initial mechanical coordinates, and the camera mechanical coordinates.

[0012] In a second aspect, the present application also provides a camera calibration device. The device comprises:

[0013] A parameter acquisition module, used to obtain the initial position parameters of the robotic arm where the camera is located at the initial working point when the center point of the target object and the plane center of the camera's image plane meet the coincidence condition; an identification module, used to perform image recognition on each rotated target object when the target object undergoes multiple rotations, and obtain the rotation angle of each rotated target object; a positioning module, used to position the robotic arm corresponding to each rotated target object based on the initial position parameters and each rotation angle, and obtain the operating position parameters of each robotic arm; a deviation acquisition module, used to obtain, for each operating position parameter, the operating working point corresponding to the operating position parameter, the position of the camera corresponding to the operating position parameter, the angular deviation between the initial working point, and the pixel coordinate deviation between the center point of the target object matched by the operating position parameter and the center of the plane; a calibration module, used to calibrate the camera based on each pixel coordinate deviation and each angular deviation.

[0014] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0015] When the center point of the target object and the center of the image plane of the camera meet the coincidence condition, the initial position parameters of the robotic arm where the camera is located at the initial working point are obtained; when the target object rotates multiple times, image recognition is performed on each rotated target object to obtain the rotation angle of each rotated target object; based on the initial position parameters and the rotation angles, the robotic arms corresponding to each rotated target object are positioned to obtain the operating position parameters of each robotic arm; for each operating position parameter, the operating working point corresponding to the operating position parameter, the position of the camera corresponding to the operating position parameter, the angle deviation between the initial working point, and the pixel coordinate deviation between the center point of the target object matched by the operating position parameter and the center of the plane are obtained; based on the pixel coordinate deviations and the angle deviations, the camera is calibrated.

[0016] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0017] When the center point of the target object and the center of the image plane of the camera meet the coincidence condition, the initial position parameters of the robotic arm where the camera is located at the initial working point are obtained; when the target object rotates multiple times, image recognition is performed on each rotated target object to obtain the rotation angle of each rotated target object; based on the initial position parameters and the rotation angles, the robotic arms corresponding to each rotated target object are positioned to obtain the operating position parameters of each robotic arm; for each operating position parameter, the operating working point corresponding to the operating position parameter, the position of the camera corresponding to the operating position parameter, the angle deviation between the initial working point, and the pixel coordinate deviation between the center point of the target object matched by the operating position parameter and the center of the plane are obtained; based on the pixel coordinate deviations and the angle deviations, the camera is calibrated.

[0018] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0019] When the center point of the target object and the center of the image plane of the camera meet the coincidence condition, the initial position parameters of the robotic arm where the camera is located at the initial working point are obtained; when the target object rotates multiple times, image recognition is performed on each rotated target object to obtain the rotation angle of each rotated target object; based on the initial position parameters and the rotation angles, the robotic arms corresponding to each rotated target object are positioned to obtain the operating position parameters of each robotic arm; for each operating position parameter, the operating working point corresponding to the operating position parameter, the position of the camera corresponding to the operating position parameter, the angle deviation between the initial working point, and the pixel coordinate deviation between the center point of the target object matched by the operating position parameter and the center of the plane are obtained; based on the pixel coordinate deviations and the angle deviations, the camera is calibrated.

[0020] The camera calibration method, device, computer equipment, storage medium and computer program product described above, when the center point of the target object and the plane center of the camera's image plane meet the coincidence condition, obtain the initial position parameters of the mechanical arm where the camera is located at the initial working point, and the center point of the target object coincides with the plane center of the camera's image plane, which helps to improve the subsequent calibration accuracy. In the case where the target object rotates multiple times, image recognition is performed on each rotated target object to obtain the rotation angle of each rotated target object, and based on the initial position parameters and each rotation angle, the position of the mechanical arm corresponding to each rotated target object is respectively located to obtain the operating position parameters of each mechanical arm, and multiple groups of mechanical arm working points are calculated, which facilitates the subsequent calibration of the camera based on multiple groups of deviations, thereby improving the camera calibration accuracy. For each operating position parameter, the operating working point corresponding to the operating position parameter, the position of the camera corresponding to the operating position parameter, the angle deviation between the initial working points, and the pixel coordinate deviation between the center point of the target object matched by the operating position parameter and the center of the plane are obtained. Using the above method, on the one hand, the camera is calibrated according to multiple groups of pixel coordinate deviations and multiple groups of angle deviations, which reduces the error generated in the calibration process and improves the camera calibration accuracy. On the other hand, the robotic arm rotates around the target object for calibration to reduce the projection transformation error at each rotation angle when the camera plane is not parallel to the object plane, thereby further improving the camera calibration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] FIG1 is a diagram of an application environment of a camera calibration method according to an embodiment;

[0022] FIG2 is a schematic diagram of a flow chart of a camera calibration method in one embodiment;

[0023] FIG3 is a schematic diagram of a flow chart of a method for correcting the visual rotation center of a collaborative robot terminal in one embodiment;

[0024] FIG4 is a schematic diagram of angle deviation and pixel deviation in one embodiment;

[0025] FIG5 is a structural block diagram of a camera calibration device in one embodiment;

[0026] FIG. 6 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0028] The camera calibration method provided in the embodiment of the present application can be applied in the application environment shown in FIG1 , wherein the robot terminal 102 communicates with the camera 104 .

[0029] When the center point of the target object and the center of the image plane of the camera 104 meet the coincidence condition, the robot terminal 102 obtains the initial position parameters of the robot arm where the camera 104 is located at the initial working point. When the target object rotates multiple times, the robot terminal 102 performs image recognition on each rotated target object to obtain the rotation angle of each rotated target object. Based on the initial position parameters and each rotation angle, the robot terminal 102 locates the robot arm corresponding to each rotated target object to obtain the operating position parameters of each robot arm. For each operating position parameter, the robot terminal 102 obtains the operating working point corresponding to the operating position parameter, the position of the camera 104 corresponding to the operating position parameter, the angular deviation between the initial working point, and the pixel coordinate deviation between the center point of the target object matching the operating position parameter and the center of the plane. The robot terminal 102 calibrates the camera 104 based on each pixel coordinate deviation and each angle deviation.

[0030] The robot terminal 102 may be, but is not limited to, a processor with image processing capabilities, such as a terminal provided by the robot. The camera 104 is a shooting device with image acquisition capabilities, such as a monocular or binocular camera.

[0031] In one embodiment, as shown in FIG2 , a camera calibration method is provided, which is described by taking the method applied to the robot terminal in FIG1 as an example, including:

[0032] S202, when the center point of the target object and the center of the image plane of the camera meet the coincidence condition, obtain the initial position parameters of the robot arm where the camera is located at the initial working point.

[0033] The target object may be a calibration object for rotation calibration of the camera. The shape and size of the target object are not limited here. The target object may be understood as an object with a specific shape and uniform thickness. For example, the target object may be a standard part in industrial production, or a paper of a specific shape.

[0034] The center point of the target object represents the physical center of the target object, for example, the center of gravity, the centroid, the center of mass, etc. Exemplarily, the center point of the target object can be called a Mark point.

[0035] The camera may be a shooting device with image acquisition capability, such as a visual camera. More specifically, the camera is used to shoot a target object to obtain image information of the target object.

[0036] The image plane of a camera refers to the imaging plane of the camera. More specifically, the image plane of a camera refers to the imaging plane in the pixel coordinate system of the camera. Similarly, the image plane may also refer to the imaging plane in the image coordinate system. The imaging plane may contain image information of a specific target object.

[0037] The plane center of the image plane of the camera represents the center point of the imaging plane of the camera in the image coordinate system or the pixel coordinate system.

[0038] The coincidence condition describes the condition corresponding to the distance between the center point of the target object and the plane center of the camera's image plane. Satisfying the coincidence condition means that the distance between the center point of the target object and the plane center of the camera's image plane satisfies the condition. More specifically, satisfying the coincidence condition means that the distance between the center point of the target object and the center point of the imaging plane in the same coordinate system type and the same imaging plane satisfies the condition. Ideally, the center point of the target object coincides with the center point of the imaging plane. Exemplarily, satisfying the coincidence condition may mean that the center point of the target object coincides with the plane center of the camera's image plane.

[0039] The robot arm refers to the end mechanism of a robot with multiple degrees of freedom. The robot arm is equipped with a camera and other fixtures. In the actual production process, the type of fixture depends on the application scenario of the robot arm and is not limited here. In the camera calibration scenario, the robot arm has the ability to rotate around the target object.

[0040] The initial working point indicates the spatial coordinate point where the robot arm is located. More specifically, the initial working point indicates the coordinate point of the robot arm in the robot arm coordinate system at the initial calibration moment.

[0041] The initial position parameters represent the coordinate information of the robot arm at the initial working point, for example, the coordinate value (x, y, z) of the robot arm in the robot arm coordinate system at the initial calibration moment.

[0042] Specifically, the robot terminal obtains the initial position parameters of the robot arm where the camera is located at the initial working point.

[0043] In one of the embodiments, the robot terminal detects the degree of overlap between the center point of the target object and the plane center of the image plane of the camera to obtain an overlap detection result. When the overlap detection result meets the overlap condition, the robot terminal obtains the initial position parameters of the robotic arm where the camera is located at the initial working point.

[0044] S204: When the target object undergoes multiple rotations, image recognition is performed on each rotated target object to obtain a rotation angle of each rotated target object.

[0045] The rotation refers to the rotation of the target object around a certain point or around a certain rotation axis.

[0046] The target object may be rotated manually or automatically, for example, based on a rotating platform.

[0047] The rotation means that the target object rotates around a certain point as the center, and the rotation can also mean that the target object rotates around a certain line as the rotation axis. Since the target object can be a calibration object, such as a calibration sticker, only the case where the target object rotates around a certain point as the center can be considered.

[0048] The angle between the rotated target and the target. The rotation angle represents the angle between the rotated target and the target. The rotation angle of each rotated target represents the angle between each rotated target and the target.

[0049] Image recognition refers to the recognition of a target object, and also refers to the recognition of a rotated target object.

[0050] Specifically, the robot terminal can control the target object to rotate multiple times, and perform image recognition on the target object after each rotation to obtain the rotation angle of each target object after each rotation. Exemplarily, the robot terminal can perform image recognition on the target object after each rotation to obtain the rotation angle corresponding to this rotation; the robot terminal can also perform image acquisition after each rotation of the target object to obtain the image of each rotation, and then, after multiple rotations, based on the images corresponding to each rotation, perform image recognition on the target object after each rotation to obtain the rotation angle of each target object after each rotation.

[0051] In one embodiment, the robot terminal can perform image recognition on the target object before it rotates to obtain image features of the target object before rotation, and perform image recognition on the rotated target object after it rotates to obtain image features of the rotated target object, and then determine the rotation angle of each rotated target object by comparing the image features corresponding to the target object before and after rotation.

[0052] In one embodiment, the robot terminal can determine the rotation center point of the target object during the rotation process, and based on the rotation center point, perform image recognition on each rotated target object to obtain the rotation angle of the rotated target object. Exemplarily, the robot terminal can determine a first line between the rotation center point and a feature point on the target object before the rotation, and a second line between the rotation center point and the feature point after the rotation, and determine the angle between the second line and the first line as the rotation angle of the rotated target object.

[0053] S206, based on the initial position parameters and the rotation angles, positioning the robot arms corresponding to the rotated target objects respectively to obtain the operating position parameters of the robot arms.

[0054] It should be noted that the purpose of position positioning is to make the target object as close to the imaging plane of the camera as possible. To achieve this goal, it is necessary to determine the camera position corresponding to the rotated target object. The camera and the manipulator are rigidly connected, and the position of the camera can be derived from the manipulator. Through the above analysis, the purpose of position positioning is to determine the manipulator position of the manipulator. The position of the manipulator can be represented by the operating position parameter.

[0055] The running position parameter indicates the coordinate information of the robot arm corresponding to the target object after rotation in the robot arm coordinate system, while the initial position parameter indicates the coordinate information of the robot arm corresponding to the target object before rotation in the robot arm coordinate system.

[0056] Specifically, for each rotated target object, the robot terminal can locate the position of the robotic arm corresponding to the rotated target object based on the initial position parameters and the rotation angle, and obtain the operating position parameters of the robotic arm.

[0057] In one of the embodiments, for each rotated target object, the robot terminal determines the rotation angle of the rotated target object as the rotation posture angle of the robotic arm. The robot terminal locates the robotic arm corresponding to the rotated target object based on the rotation posture angle and initial position parameters of the robotic arm to obtain the operating position parameters of the robotic arm.

[0058] In one embodiment, for each rotated target object, the robot terminal determines the rotation angle of the rotated target object as the rotation posture angle of the robot arm. When the robot performs image recognition on the rotated target object, it also obtains the pixel position of the center point of the target object. The robot terminal uses the pixel position of the center point of the target object as the rotation center, the distance between the pixel position and the initial position parameter as the rotation radius and the rotation angle to obtain the operating position parameters of the robot arm. Among them, the distance between the pixel position and the initial position parameter can represent the distance between the mechanical coordinates of the robot arm coordinate system corresponding to the pixel position and the mechanical coordinates corresponding to the initial position parameter.

[0059] S208, for each operating position parameter, obtain the operating working point corresponding to the operating position parameter, the position of the camera corresponding to the operating position parameter, the angle deviation between the initial working point, and the pixel coordinate deviation between the center point of the target object matched by the operating position parameter and the center of the plane.

[0060] The operating point represents the spatial coordinate point where the robot is located. More specifically, the operating point represents the coordinate point of the robot in the robot coordinate system at the time of operation calibration.

[0061] The operating position parameters may represent the coordinate information of the robot arm at the operating point, for example, the coordinate value (x1, y1, z1) of the robot arm in the robot arm coordinate system at the time of operating calibration.

[0062] The position of the camera corresponding to the operating position parameter represents the coordinate value (x2, y2, z2) of the camera in the robot coordinate system under the operating position parameter.

[0063] Exemplarily, the operating point corresponding to the operating position parameter may be represented as R, the position of the camera corresponding to the operating position parameter may be represented as C, and the initial operating point may be represented as O. The angle deviation may be ∠OCR.

[0064] The pixel coordinate deviation between the center point of the target object and the center of the plane for running position parameter matching represents the coordinate difference between the pixel coordinate of the center point of the target object and the pixel coordinate of the center of the plane.

[0065] The pixel coordinates of the center point of the target object matched by the running position parameters represent the pixel coordinates of the center point of the target object matched by the running position parameters after rotation in the pixel coordinate system under the condition of the running position parameters.

[0066] The pixel coordinates of the center of the plane represent the pixel coordinates of the center point of the imaging plane in the pixel coordinate system.

[0067] Exemplarily, the pixel coordinate deviation describes the vector deviation formed by two pixel coordinate points. For example, the pixel coordinate deviation can be a deviation vector pointing from the center of the plane to the center point of the target object, with the coordinate difference as the modulus.

[0068] Specifically, for each operating position parameter, the robot terminal obtains the operating work point corresponding to the operating position parameter, the position of the camera corresponding to the operating position parameter, the angle deviation between the initial working points, and the pixel coordinate deviation between the center point of the target object matching the operating position parameter and the center of the plane.

[0069] In one of the embodiments, with respect to the angle deviation, the robot terminal obtains the operating working point corresponding to each operating position parameter, and the robot terminal derives the position of the camera corresponding to the operating position parameter based on the operating working point, and determines the operating working point corresponding to the operating position parameter, the position of the camera corresponding to the operating position parameter, and the angle deviation between the initial working point.

[0070] In one of the embodiments, with respect to the angle deviation, the robot terminal obtains, for each operating position parameter, an operating working point corresponding to the operating position parameter, obtains the position of the camera corresponding to the operating position parameter from a position sensor corresponding to the camera, and determines the angle deviation between the operating working point corresponding to the operating position parameter, the position of the camera corresponding to the operating position parameter, and the initial working point.

[0071] S210, calibrating the camera based on the coordinate deviations of each pixel and the angle deviations.

[0072] Among them, a rotated target object corresponds to a pixel coordinate deviation and an angle deviation.

[0073] Specifically, the robot terminal calibrates the camera based on the coordinate deviation of each pixel and the angle deviation.

[0074] In one embodiment, the robot terminal maps each pixel coordinate deviation and each angle deviation based on an affine transformation matrix to obtain a mapped deviation vector, and the robot terminal calibrates the camera based on the mapped deviation vector and each angle deviation. The affine transformation matrix may be calibration data obtained by pre-calibrating the target object at nine points.

[0075] In one of the embodiments, after the camera is calibrated, the robot terminal can obtain a new set of calibration data (new coordinate deviation and new angle deviation). If the new coordinate deviation exceeds the coordinate deviation threshold, the robot terminal calibrates the camera according to the new calibration data.

[0076] In the above camera calibration method, when the center point of the target object and the plane center of the camera's image plane meet the coincidence condition, the initial position parameters of the robot arm where the camera is located at the initial working point are obtained, and the center point of the target object coincides with the plane center of the camera's image plane, which helps to improve the subsequent calibration accuracy. In the case where the target object rotates multiple times, image recognition is performed on each rotated target object to obtain the rotation angle of each rotated target object. Based on the initial position parameters and each rotation angle, the robot arm corresponding to each rotated target object is positioned to obtain the operating position parameters of each robot arm, and multiple groups of robot arm working points are calculated, which facilitates the subsequent calibration of the camera based on multiple groups of deviations, thereby improving the camera calibration accuracy. For each operating position parameter, the operating working point corresponding to the operating position parameter, the position of the camera corresponding to the operating position parameter, the angle deviation between the initial working points, and the pixel coordinate deviation between the center point of the target object matched by the operating position parameter and the center of the plane are obtained. Using the above method, on the one hand, the camera is calibrated according to multiple groups of pixel coordinate deviations and multiple groups of angle deviations, which reduces the error generated in the calibration process and improves the camera calibration accuracy. On the other hand, the robotic arm rotates around the target object for calibration to reduce the projection transformation error at each rotation angle when the camera plane is not parallel to the object plane, thereby further improving the camera calibration accuracy.

[0077] In one embodiment, image recognition is performed on each rotated target object to obtain the rotation angle of each rotated target object, including: determining the rotation center point of each rotated target object, and based on each rotation center point, performing image recognition on each rotated target object to obtain the rotation angle of each rotated target object.

[0078] The rotation center point represents the rotation center between the target object after rotation and the target object before rotation. The target object before rotation is rotated around the rotation center point to obtain the target object after rotation. Exemplarily, the target object after rotation and the target object before rotation can be regarded as markers with specific shapes and negligible thickness. Therefore, when rotating the target object, only the rotation on a single plane can be considered without considering the rotation of the target object in three-dimensional space.

[0079] Specifically, the robot terminal controls the target object to rotate, and then determines the center of the target object as the rotation center of each rotated target object. Based on the center of the target object, image recognition is performed on each rotated target object to obtain the rotation angle of each rotated target object.

[0080] In one of the embodiments, the robot terminal performs image recognition on each rotated target object based on the center of the target object to obtain image feature points of each rotated target object. For each rotated target object, the robot terminal matches the image feature points of the rotated target object with the image feature points of the target object before rotation to determine the rotation angle of the rotated target object.

[0081] In one of the embodiments, the robot terminal performs image recognition on each rotated target object based on the center of the target object in a pixel coordinate system to obtain image feature points of each rotated target object. For each rotated target object, the robot terminal matches the image feature points of the rotated target object with the image feature points of the target object before rotation (matching the pixel coordinates of each of the two image features), determines the rotation angle between the two image feature points and the center point of the target object (pixel coordinates), and determines the rotation angle as the rotation angle of the rotated target object.

[0082] In this embodiment, the rotation center point of each rotated target object is determined, and image recognition is performed on each rotated target object based on each rotation center point to obtain the rotation angle of each rotated target object. Image recognition is used to accurately determine the rotation angle of the target object without the need for visual estimation, thereby improving the accuracy of determining the rotation angle and the accuracy of subsequent camera calibration.

[0083] In one embodiment, based on each rotation center point, image recognition is performed on each rotated target object to obtain the rotation angle of each rotated target object, including: contour extraction of the target object to determine the initial contour of the target object before rotation, contour extraction of each rotated target object to obtain the target contour of each rotated target object, for each rotated target object, contour matching the initial contour with the corresponding target contour feature to obtain the contour difference between the target contour and the initial contour, and based on the contour difference corresponding to each rotated target object and each rotation center point, the rotation angle of each rotated target object is determined.

[0084] Among them, contour extraction is a specific step in image recognition. Contour extraction is aimed at the edge detection results of the image, and further connects the edge points into continuous curves or shapes to form a complete contour of the target object. The complete contour reflects the shape, area, perimeter and other information of the target object.

[0085] The initial contour of the target object before rotation represents the complete contour of the target object in the pixel coordinate system before rotation. The target contour of the target object after rotation represents the complete contour of the target object in the pixel coordinate system after rotation.

[0086] Specifically, the robot terminal extracts the contour of the target to determine the initial contour of the target before rotation, and the robot terminal extracts the contour of each rotated target to obtain the target contour of each rotated target.

[0087] Contour matching can be to match the area, perimeter, shape, minimum enclosing rectangle and other information of two complete contours.

[0088] The contour difference between the target contour and the initial contour means the difference between the perimeter of the target contour and the perimeter of the initial contour, the difference between the area of ​​the target contour and the area of ​​the initial contour, the difference between the shape of the target contour and the shape of the initial contour, and the like.

[0089] Specifically, for each rotated target object, the robot terminal matches the initial contour with the corresponding target contour feature to obtain the contour difference between the target contour and the initial contour, and determines the rotation angle of each rotated target object based on the corresponding contour difference and rotation center points of each rotated target object.

[0090] In one of the embodiments, the robot terminal performs a rotation simulation on the target before rotation based on the rotation center point for each rotated target object to obtain a simulated rotated target object. When the contour difference between the simulated rotated target object and the rotated target object meets a difference threshold, the robot terminal determines the angle of the simulated rotated target object as the rotation angle of the rotated target object.

[0091] In this embodiment, contour extraction is performed on the target object to determine the initial contour of the target object before rotation, and contour extraction is performed on each rotated target object to obtain the target contour of each rotated target object. For each rotated target object, the initial contour is matched with the corresponding target contour feature to obtain the contour difference between the target contour and the initial contour. The contour extraction method is used to determine the rotation angle of each rotated target object based on the contour difference corresponding to each rotated target object and each rotation center point, thereby improving the accuracy of the rotation angle determination.

[0092] In one embodiment, the camera is calibrated based on each pixel coordinate deviation and each angle deviation, including: obtaining an affine transformation matrix between the pixel coordinate system and the robot coordinate system, mapping each pixel coordinate deviation based on the affine transformation matrix to obtain the machine coordinate deviation of each pixel coordinate deviation in the robot coordinate system, determining the physical compensation vector of the camera based on each machine coordinate deviation and each angle deviation, and updating the calibration data of the camera based on the physical compensation vector.

[0093] The pixel coordinate system represents a coordinate system of the pixel position of the image. The origin of the pixel coordinate system may be the point where the center of the image plane of the camera is located.

[0094] The manipulator coordinate system can also be called the robot coordinate system or the world coordinate system.

[0095] The affine transformation matrix represents the transformation matrix that realizes image rotation, scaling, translation and other operations during image processing. More specifically, the affine transformation matrix can transform the pixel coordinate system and the robot coordinate system into each other. For example, the robot terminal transforms a pixel coordinate point in the pixel coordinate system into a mechanical coordinate point in the robot coordinate system.

[0096] Among them, the affine transformation matrix can be obtained by calibrating the camera at nine points.

[0097] The mechanical coordinate deviation indicates the deviation in the robot arm coordinate system. The mechanical coordinate deviation also indicates the actual distance deviation between the center point A of the target object that has not rotated and the center point B of the target object that has rotated, and the unit may be millimeters.

[0098] Among them, the physical compensation vector of the camera can represent the correction vector of the rotation calibration of the camera. The physical compensation vector can be used to update the calibration data of the camera. The calibration data of the camera includes nine-point calibration data and rotation calibration data. Calibration of the camera is essentially to update the calibration data of the camera. If the updated camera passes the calibration test, it is considered that the camera corresponding to the updated calibration data has completed the camera calibration.

[0099] Specifically, the robot terminal performs nine-point calibration on the camera to obtain the affine transformation matrix between the pixel coordinate system and the robotic arm coordinate system. Based on the affine transformation matrix, the robot terminal maps each pixel coordinate deviation respectively to obtain the machine coordinate deviation of each pixel coordinate deviation in the robotic arm coordinate system. The robot terminal determines the physical compensation vector of the camera based on each machine coordinate deviation and each angle deviation.

[0100] Specifically, the robot terminal updates the camera calibration data based on the physical compensation vector to obtain updated camera calibration data.

[0101] Furthermore, the robot terminal performs camera calibration again based on the updated camera calibration data. When the target object rotates, image recognition is performed on the rotated target object to obtain a new rotation angle of the target object. The robot terminal locates the robotic arm corresponding to the target object based on the initial position parameters and the new rotation angle to obtain new operating position parameters of the robotic arm. The robot terminal obtains a new operating working point corresponding to the new operating position parameters, the position of the camera corresponding to the new operating position parameters, the angle deviation between the initial working point, and a new pixel coordinate deviation between the center point of the target object and the center of the plane matching the new operating position parameters. When the new pixel coordinate deviation is less than a threshold, the robot terminal calibrates the camera based on each pixel coordinate deviation and each angle deviation.

[0102] In this embodiment, the affine transformation matrix between the pixel coordinate system and the robot coordinate system is obtained, and based on the affine transformation matrix, each pixel coordinate deviation is mapped respectively to obtain the machine coordinate deviation of each pixel coordinate deviation in the robot coordinate system, and based on each machine coordinate deviation and each angle deviation, the physical compensation vector of the camera is determined, and the calibration data of the camera is updated based on the physical compensation vector, thereby improving the correction accuracy of subsequent cameras.

[0103] In one embodiment, based on the affine transformation matrix, each pixel coordinate deviation is mapped separately to obtain the machine coordinate deviation of each pixel coordinate deviation in the robot arm coordinate system, including: obtaining a pixel deviation threshold configured for the target object, and when the number of pixel coordinate deviations greater than the pixel deviation threshold meets a preset value, based on the affine transformation matrix, each pixel coordinate deviation is mapped separately to obtain the machine coordinate deviation of each pixel coordinate deviation in the robot arm coordinate system.

[0104] The pixel deviation threshold represents the restriction condition configured for the pixel coordinate deviation corresponding to the target object.

[0105] If the pixel coordinate deviation is greater than the pixel deviation threshold, and a large number (greater than the preset value) of pixel coordinate deviations are greater than the pixel deviation vector, it is considered that the current camera still needs to be calibrated. If the pixel coordinate deviation is less than the pixel deviation threshold, and most of the pixel coordinate deviations are less than the pixel deviation vector, it is considered that the current camera has been preliminarily calibrated. If all pixel coordinate deviations are less than the pixel deviation vector, it can be considered that the current camera has been calibrated, and the calibration data carried by the camera does not need to be further updated.

[0106] The number of pixel coordinate deviations corresponds to the number of rotations of the target object. For example, the number of rotations of the target object may be 10 times or other times.

[0107] Specifically, the robot terminal obtains a pixel deviation threshold configured for the target object. When the number of pixel coordinate deviations greater than the pixel deviation threshold meets a preset value, the robot terminal maps each pixel coordinate deviation based on the affine transformation matrix to obtain the machine coordinate deviation of each pixel coordinate deviation in the robot arm coordinate system.

[0108] Specifically, the robot terminal obtains a pixel deviation threshold configured for the target object, and when the number of pixel coordinate deviations greater than the pixel deviation threshold does not meet a preset value, the robot terminal saves the current calibration data of the camera.

[0109] In one embodiment, the preset value includes a first preset value and a second preset value, the first preset value is greater than the second preset value, and when the number of pixel coordinate deviations greater than the pixel deviation threshold is greater than the first preset value, the robot terminal maps each pixel coordinate deviation based on the affine transformation matrix to obtain the machine coordinate deviation of each pixel coordinate deviation in the robot arm coordinate system.

[0110] When the number of pixel coordinate deviations greater than the pixel deviation threshold is greater than a second preset value and less than or equal to a first preset value, the robot terminal maps each specific pixel coordinate deviation greater than the pixel deviation threshold based on the affine transformation matrix to obtain the specific machine coordinate deviation of each characteristic pixel coordinate deviation in the robot arm coordinate system. The robot terminal then calibrates the camera based on each specific machine coordinate deviation and each angle deviation.

[0111] When the number of pixel coordinate deviations greater than the pixel deviation threshold is less than or equal to a second preset value, the robot terminal saves the current calibration data of the camera.

[0112] In this embodiment, a pixel deviation threshold configured for the target object is obtained. When the number of pixel coordinate deviations greater than the pixel deviation threshold meets the preset value, each pixel coordinate deviation is mapped based on the affine transformation matrix to obtain the machine coordinate deviation of each pixel coordinate deviation in the robot coordinate system. Only the camera in specific cases is calibrated, which reduces unnecessary camera calibration and saves hardware resources.

[0113] In one embodiment, the position of the camera includes the camera mechanical coordinates of the camera in a robot arm coordinate system;

[0114] Obtain the operating point corresponding to the operating position parameter, the position of the camera corresponding to the operating position parameter, and the angle deviation between the initial operating point, including:

[0115] Determine the spatial coordinate correspondence between the camera and the robotic arm, obtain the operating mechanical coordinates of the operating working point corresponding to the operating position parameters in the robotic arm coordinate system, and the initial mechanical coordinates of the initial working point in the robotic arm coordinate system, determine the camera mechanical coordinates based on the initial mechanical coordinates and the spatial coordinate correspondence, and determine the operating working point corresponding to the operating position parameters, the position of the camera corresponding to the operating position parameters, and the angular deviation between the initial working point based on the operating mechanical coordinates, the initial mechanical coordinates, and the camera mechanical coordinates.

[0116] The spatial coordinate correspondence between the camera and the robotic arm may represent the spatial coordinate correspondence between the camera and the robotic arm in the same coordinate system, for example, the spatial coordinate correspondence in the robotic arm coordinate system.

[0117] The operating mechanical coordinates represent the spatial coordinates of the robot arm in the robot arm coordinate system under the operating position parameters, that is, the mechanical coordinates of the operating working point.

[0118] The initial mechanical coordinates represent the mechanical coordinates of the robot at the initial working point, that is, the spatial coordinates of the robot in the robot coordinate system under the initial position parameters.

[0119] The camera mechanical coordinates represent the mechanical coordinates of the camera in the robot arm coordinate system.

[0120] Specifically, the robot terminal determines the spatial coordinate correspondence between the camera and the robotic arm, obtains the operating mechanical coordinates of the operating working point in the robotic arm coordinate system corresponding to the operating position parameters, and the initial mechanical coordinates of the initial working point in the robotic arm coordinate system.

[0121] Specifically, the robot terminal determines the camera mechanical coordinates based on the correspondence between the initial mechanical coordinates and the spatial coordinates.

[0122] In one of the embodiments, the robot terminal may also directly obtain the camera mechanical coordinates of the camera from a position positioning sensor where the camera is located.

[0123] Specifically, the robot terminal determines the operating working point corresponding to the operating position parameters, the position of the camera corresponding to the operating position parameters, and the angular deviation between the initial working points based on the operating mechanical coordinates, the initial mechanical coordinates, and the camera mechanical coordinates.

[0124] In one embodiment, the robot uses the camera mechanical coordinates as an angle, and based on the operating mechanical coordinates, the initial mechanical coordinates, and the camera mechanical coordinates, determines the operating working point corresponding to the operating position parameters, the position of the camera corresponding to the operating position parameters, and the angular deviation between the initial working point.

[0125] In this embodiment, the spatial coordinate correspondence between the camera and the robotic arm is determined, the operating mechanical coordinates of the operating working point corresponding to the operating position parameters in the robotic arm coordinate system, and the initial mechanical coordinates of the initial working point in the robotic arm coordinate system are obtained, and the camera mechanical coordinates are determined based on the initial mechanical coordinates and the spatial coordinate correspondence. Based on the operating mechanical coordinates, the initial mechanical coordinates, and the camera mechanical coordinates, the operating working point corresponding to the operating position parameters, the position of the camera corresponding to the operating position parameters, and the angular deviation between the initial working point are determined, which is beneficial for subsequent calibration of the camera according to multiple groups of angle deviations and multiple groups of pixel coordinate deviations, thereby improving the calibration accuracy of the camera.

[0126] In one embodiment, as shown in FIG3 , a method for correcting the rotation center of the end vision of a collaborative robot is provided, comprising:

[0127] S302, when the center point of the target object and the center of the image plane of the camera meet the coincidence condition, obtain the initial position parameters of the robot arm where the camera is located at the initial working point.

[0128] S304, when the target object undergoes multiple rotations, determining the rotation center points of the target objects after each rotation.

[0129] S306, extracting the contour of the target object to determine the initial contour of the target object before rotation.

[0130] S308 , extracting contours of the rotated targets respectively to obtain target contours of the rotated targets.

[0131] S310 , for each rotated target object, perform contour matching between the initial contour and the corresponding target contour feature to obtain a contour difference between the target contour and the initial contour.

[0132] S312, determining the rotation angle of each rotated target object based on the contour difference corresponding to each rotated target object and each rotation center point.

[0133] S314, based on the initial position parameters and the rotation angles, respectively locate the robot arms corresponding to the rotated target objects to obtain the operating position parameters of the robot arms.

[0134] S316, determining the spatial coordinate correspondence between the camera and the robotic arm.

[0135] S318, obtaining the operating mechanical coordinates of the operating working point in the mechanical arm coordinate system corresponding to the operating position parameter and the initial mechanical coordinates of the initial working point in the mechanical arm coordinate system.

[0136] S320, determining the camera mechanical coordinates based on the correspondence between the initial mechanical coordinates and the spatial coordinates.

[0137] S322, based on the operating mechanical coordinates, the initial mechanical coordinates, and the camera mechanical coordinates, determine the operating working point corresponding to the operating position parameter, the position of the camera corresponding to the operating position parameter, and the angular deviation between the initial working point.

[0138] As shown in Figure 4, the schematic diagram of angle deviation and pixel deviation shows that, for the robot coordinate system, RobotEnd1, RobotEnd2, RobotEnd3 and RobotEnd4 represent the operating points, i.e., the operating points of the robot under different operating position parameters. Camera1 represents the position information of the camera (Camera2, Camera3 and Camera4 are not shown in the figure), and Camera1 corresponds to RobotEnd1. Mark1 represents the center of the target object.

[0139] For the pixel coordinate system, R1 represents the pixel coordinate of the center of the target object photographed by the camera when the robot arm runs at the working point RobotEnd1, R2 represents the pixel coordinate of the center of the target object photographed by the camera when the robot arm runs at the working point RobotEnd2, and the same applies to R3 and R4.

[0140] Among them, the angle deviation ∠OCR formed by the operating working point RobotEnd1 corresponding to the operating position parameters, the position Camera1 of the camera corresponding to the operating position parameters, and the initial working point Origin / RobotEnd0, wherein O represents Origin, C represents Camera1, and R represents RobotEnd1.

[0141] S324, obtaining the pixel coordinate deviation between the center point of the target object matched with the running position parameters and the center of the plane, the affine transformation matrix between the pixel coordinate system and the robot arm coordinate system, and the pixel deviation threshold configured for the target object.

[0142] 4 , with respect to the pixel coordinate system, the pixel coordinate deviations between the center points R1 , R2 , R3 and R4 of the target object for running position parameter matching and the plane center O are OR1 , OR2 , OR3 and OR4 , respectively.

[0143] S326, when the number of pixel coordinate deviations greater than the pixel deviation threshold meets the preset value, each pixel coordinate deviation is mapped based on the affine transformation matrix to obtain the machine coordinate deviation of each pixel coordinate deviation in the robot coordinate system.

[0144] S328, determining the physical compensation vector of the camera based on the machine coordinate deviations and the angle deviations.

[0145] For example, after the target object is rotated at different angles, multiple sets of deviation vectors (OR) and deviation angles (∠OCR) can be obtained. At this time, we can map the deviation vectors back to the actual physical coordinates through the previously calibrated nine-point affine transformation matrix. However, considering the possibility that the camera plane and the object plane are not parallel, all deviation vectors need to be multiplied by a rotation vector in the Rx and Ry directions before mapping. Finally, all mapped values ​​are fitted to obtain the approximate physical compensation vector Compensate of the rotation center. The compensation is added to the calibration data, and the above fitting operation is repeated to obtain a new compensation value. Finally, when the center difference is less than a certain threshold, the correction is considered successful.

[0146] Assume the nine-point affine transformation matrix is ​​H, and the rotation matrix between the camera plane and the object plane is R, then:

[0147]

[0148] Among them, for the approximate physical compensation vector Compensate, please refer to Figure 4. Under the conventional nine-point calibration and rotation calibration, the camera plane and the object plane are required to be roughly parallel, and the lens distortion is ignored. The entire camera plane and the object plane are roughly a linear system;

[0149] It can be assumed that the rotation matrix from the camera plane to the object plane in the x and y directions is Rxy at the origin position, and the rotation in the z direction is converted from the deflection angle ∠OCR to Rz, then R-1=Rxy*Rz, and H-1 represents the nine-point affine transformation matrix.

[0150] Under RobotEnd at different positions, the deflection vector OR and the deflection angle ∠OCR are known. Since the camera plane is roughly parallel to the object plane, Rxy can be regarded as a smaller value. Rxy and Rz change with the position. With x and y randomly selected in the range of 0-10 degrees, an Rxy1 is generated, so we can get:

[0151] H-1*Rxy1*Rz1*OR1=Compensate1;

[0152] By repeatedly taking random values ​​for x and y, we can get several groups:

[0153] H-1*Rxy2*Rz2*OR2=Compensate2;

[0154] Compensate each group of the above Compensate to the original calibration data and retest. If OR shows a decreasing trend, keep this x,y value, and if it increases, discard it. Finally, several retained x,y data will be obtained, and the average value will be calculated to get the final required Compensate value.

[0155] S330: Update the calibration data of the camera based on the physical compensation vector.

[0156] In this embodiment, when the center point of the target object and the plane center of the camera's image plane meet the coincidence condition, the initial position parameters of the mechanical arm where the camera is located at the initial working point are obtained, and the center point of the target object coincides with the plane center of the camera's image plane, which helps to improve the subsequent calibration accuracy. In the case where the target object rotates multiple times, image recognition is performed on each rotated target object to obtain the rotation angle of each rotated target object. Based on the initial position parameters and each rotation angle, the mechanical arm corresponding to each rotated target object is positioned to obtain the operating position parameters of each mechanical arm, and the working points of multiple groups of mechanical arms are calculated, which facilitates the subsequent calibration of the camera based on multiple groups of deviations, thereby improving the camera calibration accuracy. For each operating position parameter, the operating working point corresponding to the operating position parameter, the position of the camera corresponding to the operating position parameter, the angle deviation between the initial working points, and the pixel coordinate deviation between the center point of the target object matched by the operating position parameter and the center of the plane are obtained. Using the above method, on the one hand, the camera is calibrated according to multiple groups of pixel coordinate deviations and multiple groups of angle deviations, which reduces the error generated in the calibration process and improves the camera calibration accuracy. On the other hand, the robotic arm rotates around the target object for calibration to reduce the projection transformation error at each rotation angle when the camera plane is not parallel to the object plane, thereby further improving the camera calibration accuracy. This method: 1. When the camera plane and the object plane are not absolutely parallel, the end-of-arm vision can fit and compensate for an error, thereby improving positioning accuracy; 2. The flexibility of nine-point calibration is retained, and high-precision calibration can be performed by positioning different templates; 3. No new hardware, such as calibration plates, etc., is required, which reduces costs; 4. The position of the robot arm is cleverly used as the actual working point when the center of the image of the end-of-arm vision camera coincides with the center of the actual detection mark point. No new hardware, such as calibration plates, calibration needles, etc., is required to correct the calibration data, which can improve the camera positioning accuracy more conveniently; 5. Multiple sets of fitting and repeated iterations are used to reduce the error caused by the robot arm posture through the camera detection data.

[0157] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0158] Based on the same inventive concept, the embodiment of the present application also provides a camera calibration device for implementing the camera calibration method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more camera calibration device embodiments provided below can refer to the limitations of the camera calibration method above, and will not be repeated here.

[0159] In one embodiment, as shown in FIG5 , a camera calibration device is provided, including: a parameter acquisition module, an identification module, a positioning module, a deviation acquisition module and a calibration module, wherein:

[0160] The parameter acquisition module 502 is used to acquire the initial position parameters of the robot arm where the camera is located at the initial working point when the center point of the target object and the center of the image plane of the camera meet the coincidence condition;

[0161] The recognition module 504 is used to perform image recognition on each rotated target object when the target object undergoes multiple rotations, and obtain the rotation angle of each rotated target object;

[0162] A positioning module 506 is used to position the robot arms corresponding to the rotated targets based on the initial position parameters and the rotation angles, and obtain the operation position parameters of the robot arms;

[0163] The deviation acquisition module 508 is used to acquire, for each operation position parameter, the operation working point corresponding to the operation position parameter, the position of the camera corresponding to the operation position parameter, the angle deviation between the initial working point, and the pixel coordinate deviation between the center point of the target object matched by the operation position parameter and the center of the plane;

[0164] The calibration module 510 is used to calibrate the camera based on the coordinate deviations of each pixel and the angle deviations.

[0165] In one embodiment, the recognition module 504 is further used to determine the rotation center point of each rotated target object; based on each rotation center point, perform image recognition on each rotated target object to obtain the rotation angle of each rotated target object.

[0166] In one embodiment, the recognition module 504 is also used to extract the contour of the target object to determine the initial contour of the target object before rotation; perform contour extraction on each rotated target object to obtain the target contour of each rotated target object; for each rotated target object, perform contour matching between the initial contour and the corresponding target contour feature to obtain the contour difference between the target contour and the initial contour; and determine the rotation angle of each rotated target object based on the contour difference corresponding to each rotated target object and each rotation center point.

[0167] In one embodiment, the calibration module 510 is also used to obtain the affine transformation matrix between the pixel coordinate system and the robot coordinate system; based on the affine transformation matrix, each pixel coordinate deviation is mapped respectively to obtain the machine coordinate deviation of each pixel coordinate deviation in the robot coordinate system; based on each machine coordinate deviation and each angle deviation, the physical compensation vector of the camera is determined; and the calibration data of the camera is updated based on the physical compensation vector.

[0168] In one embodiment, the calibration module 510 is also used to obtain a pixel deviation threshold configured for the target object; when the number of pixel coordinate deviations greater than the pixel deviation threshold meets a preset value, each pixel coordinate deviation is mapped separately based on the affine transformation matrix to obtain the machine coordinate deviation of each pixel coordinate deviation in the robot arm coordinate system.

[0169] In one embodiment, the position of the camera includes the camera mechanical coordinates of the camera in the robotic arm coordinate system; the parameter acquisition module 502 is also used to determine the spatial coordinate correspondence between the camera and the robotic arm; obtain the operating mechanical coordinates of the operating working point corresponding to the operating position parameter in the robotic arm coordinate system, and the initial mechanical coordinates of the initial working point in the robotic arm coordinate system; determine the camera mechanical coordinates based on the initial mechanical coordinates and the spatial coordinate correspondence; determine the operating working point corresponding to the operating position parameter, the position of the camera corresponding to the operating position parameter, and the angle deviation between the initial working point based on the operating mechanical coordinates, the initial mechanical coordinates, and the camera mechanical coordinates.

[0170] Each module in the above camera calibration device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0171] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG6. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store initial position parameter data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a camera calibration method is implemented.

[0172] Those skilled in the art will understand that the structure shown in FIG. 6 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0173] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0174] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0175] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0176] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0177] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0178] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A camera calibration method, characterized in that: The method comprises: When the center point of the target object and the center of the image plane of the camera meet the coincidence condition, obtaining the initial position parameters of the robot arm where the camera is located at the initial working point; In the case where the target object undergoes multiple rotations, performing image recognition on each rotated target object to obtain a rotation angle of each rotated target object; Based on the initial position parameters and the rotation angles, positioning the robot arms corresponding to the rotated targets respectively to obtain the operating position parameters of the robot arms; For each of the operating position parameters, obtaining an operating working point corresponding to the operating position parameter, a position of a camera corresponding to the operating position parameter, an angle deviation between the initial working point, and a pixel coordinate deviation between a center point of a target object matched by the operating position parameter and a center of the plane; The camera is calibrated based on the pixel coordinate deviations and the angle deviations.

2. The method according to claim 1, characterized in that The performing image recognition on each rotated target object to obtain the rotation angle of each rotated target object includes: Determine the rotation center point of each rotated target object; Based on each of the rotation center points, image recognition is performed on each of the rotated targets to obtain a rotation angle of each of the rotated targets.

3. The method according to claim 2, characterized in that The performing image recognition on each of the rotated targets based on each of the rotation center points to obtain the rotation angle of each of the rotated targets includes: Extracting the contour of the target object to determine the initial contour of the target object before rotation; Extracting contours of the rotated targets respectively to obtain respective target contours of the rotated targets; For each of the rotated targets, performing contour matching between the initial contour and the corresponding target contour feature to obtain a contour difference between the target contour and the initial contour; Based on the contour differences corresponding to the rotated targets and the rotation center points, the rotation angles of the rotated targets are determined.

4. The method according to claim 1, characterized in that Calibrate the camera based on the pixel coordinate deviations and the angle deviations, including: Get the affine transformation matrix between the pixel coordinate system and the robot arm coordinate system; Based on the affine transformation matrix, each pixel coordinate deviation is mapped respectively to obtain a machine coordinate deviation of each pixel coordinate deviation in the robot arm coordinate system; Determining a physical compensation vector of the camera based on each of the machine coordinate deviations and each of the angle deviations; The calibration data of the camera is updated based on the physical compensation vector.

5. The method according to claim 4, characterized in that The mapping of each pixel coordinate deviation based on the affine transformation matrix to obtain the machine coordinate deviation of each pixel coordinate deviation in the robot coordinate system includes: Obtaining a pixel deviation threshold configured for the target object; When the number of pixel coordinate deviations greater than the pixel deviation threshold satisfies a preset value, each pixel coordinate deviation is mapped based on the affine transformation matrix to obtain a machine coordinate deviation of each pixel coordinate deviation in the robot coordinate system.

6. The method according to claim 1, characterized in that The position of the camera includes the camera mechanical coordinates of the camera in the robot arm coordinate system; The obtaining of the operating point corresponding to the operating position parameter, the position of the camera corresponding to the operating position parameter, and the angle deviation between the initial working point include: Determine the spatial coordinate correspondence between the camera and the robotic arm; Acquire the operating mechanical coordinates of the operating working point corresponding to the operating position parameter in the mechanical arm coordinate system, and the initial mechanical coordinates of the initial working point in the mechanical arm coordinate system; Determine the camera mechanical coordinates based on the initial mechanical coordinates and the corresponding relationship between the spatial coordinates; Based on the operation mechanical coordinates, the initial mechanical coordinates, and the camera mechanical coordinates, the operation working point corresponding to the operation position parameter, the position of the camera corresponding to the operation position parameter, and the angular deviation between the initial working point are determined.

7. A camera calibration device, characterized in that: The device comprises: A parameter acquisition module, used to acquire the initial position parameters of the mechanical arm where the camera is located at the initial working point when the center point of the target object and the center of the plane of the image plane of the camera meet the coincidence condition; A recognition module, used for performing image recognition on each rotated target object when the target object undergoes multiple rotations, so as to obtain the rotation angle of each rotated target object; A positioning module, used to position the mechanical arms corresponding to the respective rotated targets based on the initial position parameters and the respective rotation angles, and obtain the operating position parameters of the respective mechanical arms; a deviation acquisition module, for acquiring, for each of the operating position parameters, an operating working point corresponding to the operating position parameter, a position of a camera corresponding to the operating position parameter, an angle deviation between the initial working point, and a pixel coordinate deviation between a center point of a target object matched by the operating position parameter and a center of the plane; A calibration module is used to calibrate the camera based on each of the pixel coordinate deviations and each of the angle deviations.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images