Camera calibration method, apparatus, electronic equipment and storage medium without angle constraints

Abstract

This application provides a camera calibration method, apparatus, electronic device, and storage medium without angular constraints, relating to the field of computer imaging. The method includes: constructing a translation calibration matrix corresponding to the camera based on multiple specified parallel movements of a motion mechanism; the multiple specified parallel movements cause the calibration object to move multiple times within the camera's field of view; controlling the motion mechanism to rotate the calibration object to an auxiliary position; controlling the camera to take a picture when the calibration object moves to the auxiliary position to obtain an auxiliary image; fitting the pixel coordinates in the auxiliary image to obtain the pixel rotation center; constructing a rotation matrix using the pixel rotation center and the angular parameters corresponding to the motion mechanism; and correcting the translation calibration matrix to obtain a generalized calibration matrix. Applying the solution provided in this application can eliminate the influence of the difference between the calibration angle and the production angle on camera calibration.

Description

Technical Field

[0001] This application relates to the field of computer image technology, and in particular to camera calibration methods, apparatus, electronic devices and storage media without angle constraints. Background Technology

[0002] With the development of industrial mechanization, more and more enterprises are using cameras combined with motion mechanisms to complete production tasks. These motion mechanisms can be robotic arms or mechanical modules. Camera calibration is a crucial preprocessing step for implementing this method. The calibration results obtained during camera calibration can be used to guide the positioning of the motion mechanism during production, by combining the camera images with the data. For example, in the manufacturing of products such as computers, tablets, mobile phones, or digital audio players, a camera calibration process needs to be performed beforehand. Then, during production, the robotic arm is controlled to complete the production of the relevant products based on the calibration results.

[0003] In related technologies, during camera calibration, the motion mechanism moves multiple times to make the calibration object move parallel to the camera multiple times. At the same time, the camera acquires multiple images containing the calibration object. Then, based on the pixel coordinates of the calibration object in the acquired images and the physical coordinates of the motion mechanism in the physical coordinate system, a nonlinear optimization method is used to find the correspondence between the pixel coordinates and the physical coordinates that minimizes the error, thereby obtaining the calibration matrix, i.e., the calibration result.

[0004] Since the calibration results obtained using related technologies are based on the calibration angle of the motion mechanism, it is necessary to ensure that the production angle of the motion mechanism remains consistent with the calibration angle during the production process to achieve accurate positioning guidance. However, in actual production, it is often difficult to maintain consistency between the production angle and the calibration angle. Therefore, the calibration results obtained using the aforementioned technologies may fail to achieve accurate positioning guidance. Both the production angle and the calibration angle of the motion mechanism are rotation angles of the motion mechanism; the difference lies in that one is the rotation angle present during the production process, and the other is the rotation angle present during the calibration process.

[0005] It is evident that eliminating the impact of the difference between the calibration angle and the production angle on camera calibration is an urgent problem to be solved. Summary of the Invention

[0006] The purpose of this application is to provide a camera calibration method, apparatus, electronic device, and storage medium without angle constraints, so as to eliminate the influence of the difference between the calibration angle and the production angle on camera calibration. The specific technical solution is as follows:

[0007] In a first aspect, embodiments of this application provide a camera calibration method without angle constraints, applied to a control device, the method comprising:

[0008] A translation calibration matrix corresponding to the camera is constructed by performing multiple specified parallel movements on the motion mechanism; wherein, the multiple specified parallel movements cause the calibration object to perform multiple parallel movements relative to the camera;

[0009] The motion mechanism is controlled to rotate from a designated position to multiple auxiliary positions so that the calibration object rotates and moves multiple times relative to the camera, and the camera is controlled to take pictures when the motion mechanism moves to each auxiliary position to obtain an auxiliary image containing the calibration object;

[0010] The pixel coordinates of the calibration object in each auxiliary image are fitted with respect to the rotation center to obtain the pixel rotation center.

[0011] A rotation matrix is ​​constructed using the pixel rotation center and the angle parameters corresponding to the motion mechanism; wherein, the angle parameters represent the angle difference between the calibration angle of the motion mechanism and the production angle which is an unknown parameter, and the rotation matrix is ​​used to represent the transformation relationship between the pixel point before and after rotation when the pixel point rotates around the pixel rotation center in the pixel coordinate system;

[0012] Using the rotation matrix, the translation calibration matrix is ​​modified to obtain a generalized calibration matrix; wherein, the specified modification is used to ensure that at any production angle of the motion mechanism, the generalized calibration matrix is ​​used as the calibration result at the same calibration angle as that production angle.

[0013] Optionally, the method further includes:

[0014] If a predetermined optimization condition is detected, the generalized calibration matrix is ​​normalized to obtain an optimized generalized calibration matrix; wherein, the predetermined optimization condition is that the physical coordinates of the designated flange of the motion mechanism are different from the physical coordinates of the gripper of the motion mechanism, and the designated flange is the flange used to connect the gripper; the normalization process is used to convert the generalized calibration matrix into a matrix when the origin of the physical coordinate system corresponding to the motion mechanism is translated to the physical rotation center; the physical rotation center is the rotation center obtained by fitting the physical coordinates of each auxiliary position in the physical coordinate system.

[0015] Optionally, when the camera is moving, the angle parameter is the target difference obtained by subtracting the production angle, which is an unknown parameter, from the calibrated angle of the motion mechanism.

[0016] When the camera is stationary, the angle parameter is the difference obtained by taking the negative of the target difference.

[0017] Optionally, constructing a rotation matrix using the pixel rotation center and the corresponding angle parameters of the motion mechanism includes:

[0018] Determine a rotation formula that characterizes the transformation relationship between two pixels; wherein the two pixels are the pixel before rotation and the pixel after rotation, and the pixel after rotation is obtained by rotating the pixel before rotation around the pixel rotation center by the angle difference represented by the angle parameter;

[0019] Based on the rotation formula, a target formula is obtained after the rotation formula is rewritten in a specified manner; wherein, the specified rewriting is: a homogeneous coordinate rewriting method that uses the pixel points before rotation and a transformation matrix to represent the pixel points after rotation.

[0020] The transformation matrix present in the target formula is determined as a rotation matrix.

[0021] Optionally, the rotation matrix is:

[0022]

[0023] Among them, M rot Let (α-β) be the rotation matrix, (α-β) represent the angle parameter, and CenterX and CenterY be the coordinate values ​​of the pixel rotation center.

[0024] Optionally, the step of using the rotation matrix to perform a specified correction process on the translation calibration matrix to obtain a generalized calibration matrix includes:

[0025] Substituting the rotation matrix and the translation calibration matrix into a predetermined correction formula yields the generalized calibration matrix; wherein the predetermined correction formula includes: M 推算 =M*(M rot ) -1 M 推算 Let M be the generalized calibration matrix, and M be the translation calibration matrix. rot ) -1 is the inverse of the rotation matrix.

[0026] Optionally, the step of constructing the translation calibration matrix corresponding to the camera based on performing multiple specified parallel movements of the motion mechanism includes:

[0027] The motion mechanism is controlled to perform multiple parallel movements so that the calibration object moves multiple times relative to the camera, and the camera is controlled to take pictures when the motion mechanism moves to each target position to obtain a target image containing the calibration object.

[0028] A translation calibration matrix is ​​generated based on the pixel coordinates of the calibration object in each target image and the physical coordinates of the motion mechanism at each target position in the physical coordinate system.

[0029] Secondly, embodiments of this application provide a camera calibration device without angle constraints, the device comprising:

[0030] The first construction module is used to construct the translation calibration matrix corresponding to the camera based on the method of performing multiple specified parallel movements on the motion mechanism; wherein, the multiple specified parallel movements cause the calibration object to perform multiple parallel movements relative to the camera;

[0031] The control module is used to control the motion mechanism to rotate from a designated position to multiple auxiliary positions so that the calibration object rotates and moves multiple times relative to the camera, and to control the camera to take pictures when the motion mechanism moves to each auxiliary position to obtain an auxiliary image containing the calibration object;

[0032] The fitting module is used to perform fitting processing on the pixel coordinates of each auxiliary image with respect to the calibration object about the rotation center, so as to obtain the pixel rotation center;

[0033] The second construction module is used to construct a rotation matrix using the pixel rotation center and the angle parameters corresponding to the motion mechanism; wherein, the angle parameters represent the angle difference between the calibration angle of the motion mechanism and the production angle which is an unknown parameter, and the rotation matrix is ​​used to represent the transformation relationship between the pixel point before and after rotation when the pixel point rotates around the pixel rotation center in the pixel coordinate system;

[0034] The correction module is used to perform a specified correction process on the translation calibration matrix using the rotation matrix to obtain a generalized calibration matrix; wherein the specified correction process is used to ensure that the generalized calibration matrix is ​​used as the calibration result at the same calibration angle as the production angle at any production angle of the motion mechanism.

[0035] Optionally, the device further includes:

[0036] An optimization module is used to normalize the generalized calibration matrix when a predetermined optimization condition is detected, to obtain an optimized generalized calibration matrix; wherein, the predetermined optimization condition is that the physical coordinates of the specified flange of the motion mechanism are different from the physical coordinates of the gripper of the motion mechanism, and the specified flange is a flange used to connect the gripper; the normalization process is used to convert the generalized calibration matrix into a matrix when the origin of the physical coordinate system corresponding to the motion mechanism is translated to the physical rotation center; the physical rotation center is a rotation center obtained by fitting the physical coordinates of each auxiliary position in the physical coordinate system.

[0037] This application also provides an electronic device, including:

[0038] Memory, used to store computer programs;

[0039] The processor is used to execute programs stored in memory to implement a camera calibration method without angular constraints.

[0040] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a camera calibration method without angular constraints.

[0041] Beneficial effects of the embodiments in this application:

[0042] The camera calibration method without angle constraints provided in this application embodiment can construct a translation calibration matrix corresponding to the camera by performing multiple specified parallel movements on the motion mechanism. Then, the motion mechanism can be controlled to move from a specified position to multiple auxiliary positions. At this time, the calibration object also rotates and moves multiple times relative to the camera. The camera is controlled to take pictures when the motion mechanism moves to the auxiliary position to obtain auxiliary images containing the calibration object. Then, the pixel coordinates of the calibration object in each auxiliary image are fitted with respect to the rotation center to obtain the pixel rotation center. Since the angle parameter corresponding to the motion mechanism represents the angle difference between the calibration angle of the motion mechanism and the production angle which is an unknown parameter, the rotation matrix constructed using the pixel rotation center and the angle parameter can be used to represent the transformation relationship between the pixel point and the pixel rotation center in the pixel coordinate system before and after rotation. Therefore, the rotation matrix can be used to perform specified correction processing on the translation calibration matrix to obtain a calibration result that can be used as the same calibration angle as the production angle at any production angle of the motion mechanism, which is the generalized calibration matrix.

[0043] Based on the above scheme, the translation calibration matrix can be used to characterize the transformation relationship between the pixel coordinate system and the physical coordinate system at the calibration angle of the motion mechanism, while the rotation matrix can be used to characterize the transformation relationship before and after a pixel rotates around its rotation center. Therefore, the rotation matrix can be used to perform specified correction processing on the translation calibration matrix to obtain a calibration result that can be used as the same calibration angle as the production angle at any production angle of the motion mechanism, which is the generalized calibration matrix. It can be seen that this scheme can eliminate the influence of the difference between the calibration angle and the production angle on camera calibration.

[0044] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0046] Figure 1 This is a schematic diagram illustrating the principle of affine transformation.

[0047] Figure 2 A field schematic diagram of the calibration control system under stationary camera conditions provided in an embodiment of this application;

[0048] Figure 3 A schematic flowchart illustrating the camera calibration method without angle constraints provided in this application embodiment;

[0049] Figure 4 A schematic flowchart illustrating the camera calibration method without angle constraints provided in this application embodiment;

[0050] Figure 5 A schematic flowchart illustrating the camera calibration method without angle constraints provided in this application embodiment;

[0051] Figure 6 A schematic flowchart illustrating the camera calibration method without angle constraints provided in this application embodiment;

[0052] Figure 7 A schematic diagram of the structure of the camera calibration device without angle constraints provided in the embodiments of this application;

[0053] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0055] To facilitate understanding of the solution, the technical terms involved in the embodiments of this application will be introduced first.

[0056] Calibration: Establishing the correspondence between the physical coordinates of the motion mechanism and the image points belonging to the calibration object through a certain algorithm.

[0057] Calibration matrix: The result of calibration is the calibration matrix.

[0058] Calibration material: A medium that can be used to complete calibration, such as calibration plates, materials, etc.

[0059] Hand-eye calibration: This involves establishing a correspondence between the physical coordinates of the flange of a motion mechanism and the pixel coordinates of image points belonging to the calibration object using a specific algorithm. For example, for a robotic arm, hand-eye calibration involves establishing a correspondence between the physical coordinates of the robotic arm flange and the pixel coordinates of the calibration object using a specific algorithm.

[0060] Translation calibration: When a motion mechanism only performs translation, the correspondence between the physical coordinates of the motion mechanism flange and the pixel coordinates of the image points belonging to the calibration object is calibrated. For example, for a robot, translation calibration refers to calibrating the correspondence between the physical coordinates of the robot flange and the pixel coordinates of the calibration object when the robot only performs translation. For industrial applications, translation calibration can be performed when the flange coordinates of the motion mechanism are equal to the gripper coordinates.

[0061] Translation and rotation calibration: When a motion mechanism translates and rotates, the correspondence between the physical coordinates of the motion mechanism flange and the pixel coordinates of the image points belonging to the calibration object is calibrated. For example, for a robot, translation and rotation calibration refers to calibrating the correspondence between the physical coordinates of the robot arm flange and the pixel coordinates of the calibration object when the robot arm translates and rotates. For industrial applications, translation and rotation calibration can be performed when the flange coordinates of the motion mechanism are not equal to the gripper coordinates.

[0062] Photo position: The physical coordinates of the flange of the motion mechanism when the camera takes a picture; for example, for a robotic arm, the photo position is the physical coordinates of the robotic arm flange when the camera takes a picture.

[0063] Calibration shooting position: During the hand-eye calibration process, the physical coordinates of the motion mechanism flange when the camera takes a picture; for example, for a robotic arm, the calibration shooting position is the physical coordinates of the robotic arm flange when the camera takes a picture during the hand-eye calibration process.

[0064] Production photo position: The physical coordinates of the moving mechanism flange when the camera takes a picture during the production process; for example, for a robot arm, the production photo position is the physical coordinates of the robot arm flange when the camera takes a picture during the production process.

[0065] Eye-in-hand: This means the camera is mounted on a motion mechanism and is in a state of camera motion. Therefore, eye-in-hand can also be called camera motion. For example, if the motion mechanism is a robotic arm, then eye-in-hand means the camera is mounted on the robotic arm and is in a state of camera motion.

[0066] Eye-to-hand: The eye is outside the hand, meaning the camera is mounted outside the motion mechanism and is in a stationary state. Therefore, eye-to-hand can also be called camera stationary. For example, for a robotic arm, eye-to-hand means the camera is mounted outside the robotic arm and is in a stationary state.

[0067] Rotation center: The center of the rotation axis when the motion mechanism rotates; for a robotic arm, the rotation center refers to the center of the rotation axis when the robotic arm rotates.

[0068] Pixel: A point on an image.

[0069] Physical point: An actual point in the three-dimensional world.

[0070] Relative calibration: Calibration of physical point coordinates as relative coordinates.

[0071] Positioning guidance: This specifically refers to camera photography, using vision to guide robotic arms or mechanical modules to perform operations such as grasping, posture correction, and fitting.

[0072] Affine transformation: Affine transformation is an important transformation in a two-dimensional plane. Affine transformation can include rotation, translation, and scaling. Affine transformation does not change the parallel relationship of lines.

[0073] To better understand affine transformations, the following will combine... Figure 1 This section introduces affine transformations.

[0074] like Figure 1 As shown, the coordinates of point P in the original coordinate system are (X, Y). It is rotated counterclockwise by θ around the origin, with a scaling factor of f. x f yThe actual coordinates after moving (a,b) are (X′,Y′):

[0075]

[0076] The transformation matrix, expressed in homogeneous coordinate form, is as follows:

[0077]

[0078] Among them, matrix It can be used to characterize the correspondence between the original coordinate system and the new coordinate system obtained after translation and rotation.

[0079] To better understand the embodiments of this application, the prior art will be described below.

[0080] In related technologies, during camera calibration, a calibration matrix can be generated at the calibration angle of the motion mechanism to characterize the correspondence between the pixel coordinates of the calibrated object and the physical coordinates of the motion mechanism. In actual production, based on the image of the product to be controlled captured by the camera, the pixel coordinates of the product to be controlled in the pixel coordinate system are obtained. The calibration matrix obtained through the calibration process is used to convert the pixel coordinates into the physical coordinates of the motion mechanism in the physical coordinate system. The actual position of the product to be controlled can be located based on the physical coordinates of the motion mechanism. Then, based on the actual position of the product to be controlled, the motion mechanism can be controlled to guide the position of the product to be controlled, thereby completing the production process.

[0081] However, when the production angle is inconsistent with the calibration angle, the calibration matrix obtained using the aforementioned related technologies cannot achieve accurate positioning guidance.

[0082] Furthermore, in actual production, constraints in the physical structure required for production may prevent the production angle from matching the calibration angle. Therefore, during calibration, relevant technologies can use the production angle from the actual production process as the calibration angle to generate a calibration matrix for positioning guidance. However, whenever the production angle changes, the calibration angle needs to be changed, and the calibration process needs to be repeated, increasing the complexity of calibration, introducing additional errors into the calibration results, and reducing the accuracy of calibration.

[0083] As described above, related technologies utilize calibration results obtained through a calibration process to guide the positioning of motion mechanisms. However, these calibration results are applicable to the calibration angles of the motion mechanisms. When the production angle differs from the calibration angle, the calibration results obtained through the calibration process cannot be used for positioning guidance. Therefore, how to eliminate the impact of the difference between the calibration angle and the production angle on camera calibration is a problem that urgently needs to be solved.

[0084] To eliminate the impact of the difference between the calibration angle and the production angle on camera calibration, embodiments of this application provide a camera calibration method, apparatus, electronic device, and storage medium without angle constraints.

[0085] The following section first introduces a camera calibration method without angle constraints provided in the embodiments of this application.

[0086] The camera calibration method without angle constraints provided in this application can be applied to control devices. In specific applications, the control device, camera, and motion mechanism can constitute a calibration control system. The camera can be used to photograph the calibration object and send the image containing the calibration object to the control device. The motion mechanism can be used to move the calibration object or camera. The control device can control the movement of the motion mechanism and the camera to take pictures, receive the images sent by the camera, obtain the physical coordinates of the motion mechanism, and determine the calibration result based on the obtained information. Where the motion mechanism flange and the gripper used to connect the calibration object or camera are coaxial, the physical coordinates of the motion mechanism flange can be the physical coordinates of the motion mechanism. Therefore, the control device can obtain the physical coordinates of the motion mechanism by obtaining the physical coordinates of the motion mechanism flange. Depending on the method of connecting the calibration object or camera, the gripper in the motion mechanism can be a clamp or a suction nozzle; the specific form of the gripper is not limited and is only provided as an example.

[0087] It should be noted that when the camera is moving, the gripper of the motion mechanism can be connected to the camera; when the camera is stationary, the gripper of the motion mechanism can be connected to a calibration object.

[0088] For example, such as Figure 2 As shown, in the calibration control system where the motion mechanism is a robotic arm, camera 1 can be used to photograph the calibration object 2 and send the image containing the calibration object 2 to the control device; the control device can control the movement of flange 3 and gripper 4 connected to calibration object 2, thereby driving calibration object 2 to move; the control device can also control the movement of flange 3 and camera 1 to take pictures, receive the image sent by camera 1, obtain the physical coordinates of flange 3 in the physical coordinate system, and determine the calibration result based on the obtained content; wherein, flange 3 and gripper 4 are coaxial, therefore, the physical coordinates of flange 3 obtained by the control device can be the physical coordinates of the motion mechanism.

[0089] One embodiment of this application provides a camera calibration method without angle constraints, which may include the following steps:

[0090] A translation calibration matrix corresponding to the camera is constructed by performing multiple specified parallel movements on the motion mechanism; wherein, the multiple specified parallel movements cause the calibration object to perform multiple parallel movements relative to the camera;

[0091] The motion mechanism is controlled to rotate from a designated position to multiple auxiliary positions so that the calibration object rotates and moves multiple times relative to the camera, and the camera is controlled to take pictures when the motion mechanism moves to each auxiliary position to obtain an auxiliary image containing the calibration object;

[0092] The pixel coordinates of the calibration object in each auxiliary image are fitted with respect to the rotation center to obtain the pixel rotation center.

[0093] A rotation matrix is ​​constructed using the pixel rotation center and the angle parameters corresponding to the motion mechanism; wherein, the angle parameters represent the angle difference between the calibration angle of the motion mechanism and the production angle which is an unknown parameter, and the rotation matrix is ​​used to represent the transformation relationship between the pixel point before and after rotation when the pixel point rotates around the pixel rotation center in the pixel coordinate system;

[0094] Using the rotation matrix, the translation calibration matrix is ​​modified to obtain a generalized calibration matrix; wherein, the specified modification is used to ensure that at any production angle of the motion mechanism, the generalized calibration matrix is ​​used as the calibration result at the same calibration angle as that production angle.

[0095] Based on the above scheme, the translation calibration matrix can be used to characterize the transformation relationship between the pixel coordinate system and the physical coordinate system at the calibration angle of the motion mechanism, while the rotation matrix can be used to characterize the transformation relationship before and after a pixel rotates around its rotation center. Therefore, the rotation matrix can be used to perform specified correction processing on the translation calibration matrix to obtain a calibration result that can be used as the same calibration angle as the production angle at any production angle of the motion mechanism, which is the generalized calibration matrix. It can be seen that this scheme can eliminate the influence of the difference between the calibration angle and the production angle on camera calibration.

[0096] The following describes a camera calibration method without angle constraints provided by an embodiment of this application, with reference to the accompanying drawings.

[0097] Figure 3 A flowchart illustrating an angle-constrained camera calibration method provided in this application embodiment is shown below. Figure 3 As shown, the method may include the following steps:

[0098] S301, Based on the method of performing multiple specified parallel movements on the motion mechanism, a translation calibration matrix corresponding to the camera is constructed; wherein, the multiple specified parallel movements cause the calibration object to perform multiple parallel movements relative to the camera.

[0099] It is understood that the control device can send control signals to the motion mechanism to control the motion mechanism to perform parallel movement. Simultaneously, the calibration object can perform multiple parallel movements relative to the camera. The construction of the translation calibration matrix corresponding to the camera can be a hand-eye calibration process, which will be described below. The calibration object can be an intermediary used to determine the transformation relationship between pixel coordinates and physical coordinates. For example, the calibration object can be a checkerboard calibration board, a QR code calibration board, or other materials. This application embodiment does not limit the specific form of the calibration object. Regarding the multiple parallel movements of the calibration object relative to the camera, when the camera is moving, the camera can move while the calibration object remains stationary; conversely, when the camera is stationary, the calibration object can move while the camera remains stationary. Therefore, in general, the calibration object can perform parallel movements relative to the camera while the motion mechanism is moving in parallel.

[0100] In one implementation, constructing the translation calibration matrix corresponding to the camera based on multiple specified parallel movements of the motion mechanism may include steps A1-A2:

[0101] A1, control the motion mechanism to perform multiple parallel movements so that the calibration object moves multiple times relative to the camera, and control the camera to take pictures when the motion mechanism moves to each target position, so as to obtain a target image containing the calibration object.

[0102] It is understandable that the motion mechanism can reach multiple target positions by making multiple parallel movements. At the same time, the calibration object can also make multiple parallel movements relative to the camera. The multiple target positions can be the positions after the motion mechanism has made at least three non-collinear movements. Furthermore, the control device can send control signals to the camera to control the camera to take pictures when the motion mechanism moves to each target position. The camera can send the target images of the calibration object it has taken to the control device.

[0103] Regarding the movement of a motion mechanism, the motion mechanism can be driven by a control device to move a fixed distance each time to reach a target position. The fixed distance can be a displacement in the forward, backward, left, and right directions on the same plane. For example, on the plane of the (x,y) two-dimensional coordinate system, moving a certain motion mechanism along the positive x-axis by one unit distance can be called a fixed distance.

[0104] A2 generates a translation calibration matrix based on the pixel coordinates of the calibration object in each target image and the physical coordinates of the motion mechanism at each target position in the physical coordinate system.

[0105] Understandably, the control device can acquire the physical coordinates of the motion mechanism's flange during its movement. When the flange and the gripper used to connect the calibration object or camera are coaxial, the flange's physical coordinates can characterize the motion mechanism's physical coordinates. Therefore, the control device can determine the motion mechanism's physical coordinates. It should be noted that each target position can be the motion mechanism's position in the three-dimensional world, specifically represented by a three-dimensional coordinate system. However, during calibration, the motion mechanism only undergoes parallel movement; that is, one dimension of the motion mechanism in the three-dimensional world can be fixed. Therefore, the motion mechanism's physical coordinates can be two-dimensional coordinates.

[0106] To determine the pixel coordinates of the calibration object, an actual point existing in physical space can be selected on the calibration object as an anchor point to locate the calibration object in the image. After acquiring each target image, the actual point in each target image containing the actual point of the calibration object can be determined as an image point, and the coordinates of the image point can be used as the pixel coordinates of the calibration object. At the same time, in order to obtain complete and accurate pixel coordinates, the selected image point can be a point in an image with clear pixels and definite coordinate values.

[0107] To generate the translation calibration matrix, the transformation relationship between the pixel coordinates of the calibration object and the physical coordinates of the moving mechanism can be found based on the principle of affine transformation. Thus, the transformation relationship between the pixel coordinate system where the pixel coordinates are located and the physical coordinate system where the physical coordinates are located can be determined. This transformation relationship can be in the form of a matrix, that is, a translation transformation matrix.

[0108] S302, control the motion mechanism to rotate from a designated position to multiple auxiliary positions so that the calibration object rotates and moves multiple times relative to the camera, and control the camera to take pictures when the motion mechanism moves to each auxiliary position to obtain an auxiliary image containing the calibration object.

[0109] Understandably, the control device can send control signals to the motion mechanism to control the motion mechanism to rotate and move, while the calibration object can rotate and move relative to the camera; multiple auxiliary positions can be the positions after the motion mechanism has rotated at least three times; the control device in the calibration control system can send control signals to the camera to control the camera to take pictures when the motion mechanism moves to each auxiliary position; the camera can send the captured auxiliary images of the calibration object to the control device.

[0110] It should be noted that the rotation of the motion mechanism occurs in a three-dimensional world. Therefore, after multiple rotations, the movement of the motion mechanism from a designated position to an auxiliary position can be a process of changing the three-dimensional coordinates of the motion mechanism in the three-dimensional world. For the process of rotating from a designated position to multiple auxiliary positions, the calibration angle can be changed while keeping the two-dimensional physical coordinates of the designated position unchanged, thus achieving the purpose of rotating to multiple auxiliary positions. It can be understood that the motion mechanism can use a designated position as the rotation center for multiple rotational movements, and these multiple rotational movements can be three or more. The designated position can be a position specified by the control device. For example, to reduce the movement variables, the position before the motion mechanism performs parallel movement can be used as the designated position; or, to reduce the constraints of the physical structure of the motion mechanism, an open space can be used as the designated position. The designated position in this application is merely an example and is not intended to limit the scope of the application.

[0111] S303, perform a fitting process on the pixel coordinates of the calibration object in each auxiliary image with respect to the rotation center to obtain the pixel rotation center.

[0112] It is understandable that the actual points in the auxiliary images can be used as image points, and the coordinates of the image points can be used as the pixel coordinates of the calibration object. Since pixel coordinates can be obtained from each auxiliary image, and there are at least three auxiliary images, at least three pixel coordinates can be obtained.

[0113] For fitting the rotation center, a circle can be fitted using multiple point coordinates, and the center of this circle is the rotation center. Since the point coordinates used for fitting can be pixel coordinates, the center of the resulting fitted circle can be the pixel rotation center.

[0114] S304, using the pixel rotation center and the angle parameters corresponding to the motion mechanism, a rotation matrix is ​​constructed; wherein, the angle parameters represent the angle difference between the calibration angle of the motion mechanism and the production angle which is an unknown parameter, and the rotation matrix is ​​used to represent the transformation relationship between before and after the pixel point rotates around the pixel rotation center in the pixel coordinate system.

[0115] It is understandable that the angle parameter can characterize the angle difference between the calibration angle and the production angle, and the specific value of the production angle can be characterized by an unknown parameter; the rotation matrix can be used to characterize the transformation relationship of a pixel before and after rotation in the pixel coordinate system, and the rotation of a pixel can be a movement around the pixel rotation center.

[0116] Optionally, in one implementation, the rotation matrix is:

[0117]

[0118] Among them, M rot Let (α-β) be the rotation matrix, (α-β) represent the angle parameter, and CenterX and CenterY be the coordinate values ​​of the pixel rotation center.

[0119] It is understood that CenterX and CenterY are the coordinate values ​​of the pixel rotation center, which can be obtained by the fitting process of S304; (α-β) can characterize the angle parameter, and the angle parameter can characterize the angle difference between the calibration angle and the production angle; therefore, α can be the calibration angle, and correspondingly, β can be an unknown parameter characterizing the production angle; or, α can be an unknown parameter characterizing the production angle, and correspondingly, β can be the calibration angle.

[0120] It should be noted that the definitions of α and β can differ in different scenarios, and the method for calculating the angle difference can also differ. In another implementation, the specific method for calculating the angle difference may include:

[0121] B1, when the camera is moving, the angle parameter is the target difference obtained by subtracting the production angle, which is an unknown parameter, from the calibrated angle of the motion mechanism.

[0122] Understandably, when the camera is in motion, it can be mounted on a motion mechanism and move along with it; this state can be described as "eye in hand." For example, when the camera is in motion, it can be mounted on a robotic arm. In this case, α can be the calibration angle, β can be an unknown parameter representing the production angle, and (α-β) can represent the target difference obtained by subtracting the production angle (which is an unknown parameter) from the calibration angle.

[0123] B2, when the camera is stationary, the angle parameter is the difference obtained by taking the negative of the target difference.

[0124] Understandably, when the camera is stationary, it can be mounted in a fixed position outside the motion mechanism and will not move with the motion mechanism. This state can be described as "eye-to-hand." For example, when the camera is stationary, it can be mounted outside the robotic arm. In this case, the target difference obtained from B1 can be negatively taken to obtain the angle parameter.

[0125] In one implementation, constructing the rotation matrix using the pixel rotation center and the angle parameters corresponding to the motion mechanism may include the following steps:

[0126] C1, determine the rotation formula that characterizes the transformation relationship between two pixels; wherein, the two pixels are the pixel before rotation and the pixel after rotation, and the pixel after rotation is obtained by rotating the pixel before rotation around the pixel rotation center by the angle difference represented by the angle parameter.

[0127] It is understandable that the pixel before rotation can be rotated around the pixel rotation center by the angle difference represented by the angle parameter to obtain the pixel after rotation. Therefore, the correspondence between the pixel coordinates before rotation and the pixel coordinates after rotation can be represented by a set of equations containing the angle parameter around the pixel rotation center. The set of equations can be written in matrix form, which can be the rotation formula.

[0128] For example, the pixel coordinates of the calibration object before rotation are (X,Y), and the pixel coordinates after rotation are (X1,Y1). (α-β) represents the angle parameter, and the coordinates of the pixel rotation center are (CenterX,CenterY). The system of equations that can represent the correspondence between the pixel coordinates before and after rotation is as follows:

[0129]

[0130] Rewriting the above system of equations in matrix form, we obtain the rotation formula:

[0131]

[0132] C2, based on the rotation formula, obtain the target formula obtained by rewriting the rotation formula in a specified manner; wherein, the specified rewriting is: a homogeneous coordinate rewriting method that uses the pixel points before rotation and a transformation matrix to represent the pixel points after rotation.

[0133] For example, when the rotation formula is When this happens, the rotation matrix can be rewritten in the form of a homogeneous rotation matrix: The target formula is obtained.

[0134] C3, determine the transformation matrix present in the target formula as a rotation matrix.

[0135] For example, when the target formula is At that time, it can be It is determined to be a rotation matrix.

[0136] S305, using the rotation matrix, the translation calibration matrix is ​​subjected to a specified correction process to obtain a generalized calibration matrix; wherein, the specified correction process is used to ensure that at any production angle of the motion mechanism, the generalized calibration matrix is ​​used as the calibration result at the same calibration angle as that production angle.

[0137] It is understandable that the rotation matrix can represent the correspondence between the pixel coordinates of the calibration object before and after rotation, and the translation calibration matrix can represent the correspondence between the pixel coordinates of the calibration object and the physical coordinates of the motion mechanism at the calibration angle. By using the rotation matrix to perform specified correction processing on the translation calibration matrix, the rotation angle can be incorporated into the calibration result, resulting in a generalized calibration matrix that can be applied to any production angle of the motion mechanism. This generalized calibration matrix can represent the correspondence between the pixel coordinates of the calibration object in the pixel coordinate system and the physical coordinates in the physical coordinate system.

[0138] In one implementation, the step of using the rotation matrix to perform a specified correction process on the translation calibration matrix to obtain a generalized calibration matrix includes:

[0139] Substituting the rotation matrix and the translation calibration matrix into a predetermined correction formula yields the generalized calibration matrix; wherein the predetermined correction formula includes: M 推算 =M*(M rot ) -1 M 推算 Let M be the generalized calibration matrix, and M be the translation calibration matrix. rot ) -1 is the inverse of the rotation matrix.

[0140] It is understandable that before the calibration object rotates relative to the camera, the pixel coordinates on the image are (X, Y), and the physical coordinates of the motion mechanism are (X′, Y′). The translation calibration matrix M is used to characterize the correspondence between the pixel coordinates of the calibration object and the physical coordinates of the motion mechanism at the calibration angle before rotation. After the calibration object rotates relative to the camera, the pixel coordinates on the image are (X1, Y1), and the physical coordinates of the motion mechanism are (X′1, Y′1). The calibration matrix M1 is used to characterize the correspondence between the pixel coordinates of the calibration object and the physical coordinates of the motion mechanism after rotation. Therefore, M, M1, and the correspondence between the pixel coordinates and physical coordinates before and after rotation can be expressed by a system of equations. The specific formula is as follows: Since the calibration object is only rotated relative to the camera and not translated, the physical coordinates of the two-dimensional motion mechanism can remain unchanged. That is, the physical coordinates (X′, Y′) before rotation are the same as the physical coordinates (X′1, Y′1) after rotation. Accordingly, the above formula can be rewritten as M*(X, Y) = M1*(X1, Y1). The translation calibration matrix M before rotation can be obtained through prior hand-eye calibration, and the pixel coordinates before and after rotation can be obtained from the image captured by the camera. Therefore, the calibration matrix M1 after rotation can be solved. M1 can be used to characterize the transformation relationship between before and after rotation when a pixel rotates around its rotation center in the pixel coordinate system. 推算 .

[0141] Based on the above scheme, the translation calibration matrix can be used to characterize the transformation relationship between the pixel coordinate system and the physical coordinate system at the calibration angle of the motion mechanism, while the rotation matrix can be used to characterize the transformation relationship before and after a pixel rotates around its rotation center. Therefore, the rotation matrix can be used to perform specified correction processing on the translation calibration matrix to obtain a calibration result that can be used as the same calibration angle as the production angle at any production angle of the motion mechanism, which is the generalized calibration matrix. It can be seen that this scheme can eliminate the influence of the difference between the calibration angle and the production angle on camera calibration.

[0142] Furthermore, in this embodiment, the calibration angle can be deconstrained by constructing a rotation matrix based on angle parameters, eliminating the difference between the calibration angle and the production angle, and reducing the structural constraints on the physical mechanism to a certain extent. Combining the rotation matrix with the translation calibration matrix generates a generalized calibration matrix, simplifying the calibration process when the calibration shooting position angle and the production shooting position angle are inconsistent. A generalized calibration matrix that can be derived for each production angle can be obtained through a single calibration process, expanding the application boundary of traditional hand-eye calibration. This embodiment is also applicable when the production angle and the calibration angle are the same, enhancing the interpretability of traditional hand-eye calibration methods.

[0143] Optionally, in another embodiment, in Figure 3 Based on the angle-constraint-free camera calibration method shown, the following steps can also be included:

[0144] If a predetermined optimization condition is detected, the generalized calibration matrix is ​​normalized to obtain an optimized generalized calibration matrix; wherein, the predetermined optimization condition is that the physical coordinates of the designated flange of the motion mechanism are different from the physical coordinates of the gripper of the motion mechanism, and the designated flange is the flange used to connect the gripper; the normalization process is used to convert the generalized calibration matrix into a matrix when the origin of the physical coordinate system corresponding to the motion mechanism is translated to the physical rotation center; the physical rotation center is the rotation center obtained by fitting the physical coordinates of each auxiliary position in the physical coordinate system.

[0145] Understandably, the generalized calibration matrix can be used when the physical coordinates of the specified flange of the motion mechanism are the same as the physical coordinates of the gripper of the motion mechanism, that is, the physical coordinates of the specified flange can be the physical coordinates of the motion mechanism, representing the correspondence between the pixel coordinate system and the physical coordinate system at any production angle of the motion mechanism. However, when the physical coordinates of the specified flange of the motion mechanism are different from the physical coordinates of the gripper of the motion mechanism, that is, when the physical coordinates of the specified flange are different from the physical coordinates of the motion mechanism, if the physical coordinates of the specified flange are still used as the physical coordinates of the motion mechanism, it may cause the calibration to fail. Since the calibration results may produce errors, the generalized calibration matrix can be normalized. Normalization can be used to translate the origin of the physical coordinate system corresponding to the motion mechanism to the physical rotation center obtained after the calibration object is rotated, that is, to transform the coordinate values ​​of the physical rotation center to (0,0). The optimized generalized calibration matrix can eliminate possible errors when the physical coordinates of the specified flange of the motion mechanism are different from the physical coordinates of the gripper of the motion mechanism, and characterize the correspondence between the pixel coordinate system and the physical coordinate system at any production angle of the motion mechanism. The physical rotation center can be obtained by fitting the physical coordinates of the auxiliary position.

[0146] For example, the translation component in the calibration matrix can be obtained by multiplying the calibration matrix by the coordinates of the pixel rotation center until zero, thus obtaining the normalized calibration matrix. For instance, the generalized calibration matrix is... Where γ is the angle of counterclockwise rotation of the motion mechanism about the origin of the coordinate system, and g x g y The scaling factors for this generalized calibration matrix are c and d, which are the displacements of the motion mechanism in parallel movement. c and d can also be called translation components. The coordinates of the pixel rotation center are (CenterX, CenterY). After normalizing the generalized calibration matrix, the translation components c and d can be obtained. Therefore, the normalized calibration matrix is:

[0147]

[0148] In this embodiment, for cases where the physical coordinates of the specified flange of the motion mechanism are different from the physical coordinates of the gripper of the motion mechanism, an optimization scheme for a generalized calibration matrix that can characterize the correspondence between the pixel coordinate system and the physical coordinate system at any production angle of the motion mechanism is provided, increasing the application scenarios of camera calibration without angle constraints.

[0149] To better understand the camera calibration method without angle constraints provided in the embodiments of this application, another embodiment will be described below.

[0150] In this embodiment, the motion mechanism can be a robotic arm. This embodiment is only for illustrative purposes and does not limit the specific form of the motion mechanism.

[0151] Figure 4 This is a flowchart illustrating the angle-constrained camera calibration method provided in an embodiment of this application, as shown below. Figure 4 As shown, the method may include the following steps:

[0152] S401, a translation calibration matrix is ​​generated through translation calibration processing; wherein, the translation calibration matrix is ​​used to characterize the transformation relationship between the pixel coordinate system corresponding to the camera and the physical coordinate system under the calibration angle of the motion mechanism.

[0153] It is understandable that translation calibration processing can be used to calibrate the correspondence between the robot arm flange coordinates and image points when the robot arm only performs translation. Specifically, translation calibration processing can be conventional hand-eye calibration, that is, establishing the correspondence between the robot arm flange coordinates and its image points through a certain algorithm.

[0154] The transformation relationship between the pixel coordinate system corresponding to the camera and the physical coordinate system can be understood as the correspondence between the pixel coordinates of the calibration object and the physical coordinates of the motion mechanism after parallel translation.

[0155] S402, obtain the angle difference of the shooting position; wherein, the shooting position is the physical coordinate of the robot flange when the camera takes a picture, and the angle difference is the difference between the calibration angle of the motion mechanism and the production angle which belongs to the unknown parameter.

[0156] It is understandable that the camera position can include a calibration camera position and a production camera position; the calibration camera position can be the physical coordinates of the robotic arm flange when the camera takes a picture during the calibration process; the production camera position can be the physical coordinates of the robotic arm flange when the camera takes a picture during the production process. The physical coordinates of the robotic arm flange when the camera takes a picture can be the physical coordinates of the motion mechanism.

[0157] S403, construct a rotation matrix using the pixel rotation center and the angle difference; wherein, the pixel rotation center is a rotation center obtained by translation and rotation calibration.

[0158] Understandably, the rotation center can be the center of the rotation axis when the robot rotates, and the rotation center can be obtained through fitting. Specifically, a circle can be determined using multiple points, and then the center of that circle can be found, which can be the rotation center. In particular, a pixel can be a point on the image belonging to the pixel coordinate system, and a physical point can be an actual point belonging to the physical coordinate system. The rotation center determined using the pixel can be used as the pixel rotation center, and the rotation center determined using the physical point can be used as the physical rotation center.

[0159] S404, using the rotation matrix, perform a specified correction process on the translation calibration matrix to obtain a generalized calibration matrix.

[0160] It should be noted that S401 corresponds to S301-S302 in the previous embodiments, S403 corresponds to S305 in the previous embodiments, and S404 corresponds to S306 in the previous embodiments. Most of the steps in this embodiment have been described in the previous embodiments, so they will not be repeated here.

[0161] Based on the above scheme, the translation calibration matrix can be used to characterize the transformation relationship between the pixel coordinate system and the physical coordinate system at the calibration angle of the motion mechanism, while the rotation matrix can be used to characterize the transformation relationship before and after a pixel rotates around its rotation center. Therefore, the rotation matrix can be used to perform specified correction processing on the translation calibration matrix to obtain a calibration result that can be used as the same calibration angle as the production angle at any production angle of the motion mechanism, which is the generalized calibration matrix. It can be seen that this scheme can eliminate the influence of the difference between the calibration angle and the production angle on camera calibration.

[0162] Optionally, in another embodiment, such as Figure 5 As shown, in Figure 4 Based on the angle-constraint-free camera calibration method shown, the step of generating a translation calibration matrix through translation calibration processing may include the following steps:

[0163] S4011, control the robotic arm to perform multiple parallel movements to move the motion mechanism to each target position by a fixed step size each time, and control the camera to take pictures when the motion mechanism moves to each target position, so as to obtain a target image containing the calibration object.

[0164] It is understandable that the fixed step size can be the distance that the robotic arm can reach within the camera's field of view; each parallel movement is calculated to reduce the amount of translation by using the fixed step size; the fixed step size can also be called a fixed distance.

[0165] S4012, acquire the target image, extract features from the target image, and obtain N sets of translated pixel coordinates and physical coordinates; wherein, N is a natural number greater than or equal to 3.

[0166] It is understandable that extracting features from a target image can involve finding stable image points in the image and obtaining the pixel coordinates of those points; the acquisition of physical coordinates can be obtained by the control device that controls the movement of the robot during the control process. For example, in the (x,y) physical coordinate system, if the robot moves one unit displacement in the x direction from (0,0), then the physical coordinates at this time are (1,0).

[0167] S4013, Generate a translation calibration matrix based on the N sets of translated pixel coordinates and physical coordinates.

[0168] After obtaining the angle difference of the shooting position, the process may further include step S501: determining whether the angle difference is zero; if yes, the process ends; if no, step S403 is executed.

[0169] It is understood that most of the steps in this embodiment have been described in the foregoing embodiments, and will not be repeated here.

[0170] Based on the above scheme, the translation calibration matrix can be used to characterize the transformation relationship between the pixel coordinate system and the physical coordinate system at the calibration angle of the motion mechanism, while the rotation matrix can be used to characterize the transformation relationship before and after a pixel rotates around its rotation center. Therefore, the rotation matrix can be used to perform specified correction processing on the translation calibration matrix to obtain a calibration result that can be used as the same calibration angle as the production angle at any production angle of the motion mechanism, which is the generalized calibration matrix. It can be seen that this scheme can eliminate the influence of the difference between the calibration angle and the production angle on camera calibration.

[0171] Optionally, in another embodiment, such as Figure 6 As shown, in Figure 5 Based on the angle-constraint-free camera calibration method shown, before obtaining the angle difference of the shooting position, the following steps may also be included:

[0172] S601, control the robot arm to rotate multiple times to move the motion mechanism to each auxiliary position in a fixed direction each time, and control the camera to take pictures when the motion mechanism moves to each auxiliary position to obtain an auxiliary image containing the calibration object.

[0173] It is understandable that the fixed direction can be the distance that the robotic arm can move within the camera's field of view; rotating in a fixed direction each time can reduce the calculation of the amount of rotation.

[0174] S602, acquire the auxiliary image, extract features from the auxiliary image, and obtain N rotated pixel coordinates.

[0175] S603, perform fitting processing on the N rotated pixel coordinates to obtain the pixel rotation center.

[0176] After using the rotation matrix to perform a specified correction process on the translation calibration matrix to obtain the generalized calibration matrix, the process may further include step S604: normalizing the generalized calibration matrix to obtain the optimized generalized calibration matrix.

[0177] The step of determining whether the angle difference is zero; if yes, the process ends; if no, step S403 is executed, which may include step S5011: determining whether the angle difference is zero. If yes, step S605 is executed; if no, step S403 is executed.

[0178] Step S605: Normalize the translation calibration matrix to obtain the optimized translation calibration matrix.

[0179] It should be noted that normalization can eliminate potential errors when the physical coordinates of the robotic arm flange and the physical coordinates of the motion mechanism are inconsistent.

[0180] It is understood that most of the steps in this embodiment have been described in the foregoing embodiments, and will not be repeated here.

[0181] It is understood that this embodiment is based on Figure 5 The embodiments described herein, and Figure 5 Is Figure 4 The description is based on the embodiments, therefore, in Figure 4 Based on the embodiments described herein, the camera calibration method without angle constraints may also include the steps added in this embodiment.

[0182] Based on the above scheme, the translation calibration matrix can be used to characterize the transformation relationship between the pixel coordinate system and the physical coordinate system at the calibration angle of the motion mechanism, while the rotation matrix can be used to characterize the transformation relationship before and after a pixel rotates around its rotation center. Therefore, the rotation matrix can be used to perform specified correction processing on the translation calibration matrix to obtain a calibration result that can be used as the same calibration angle as the production angle at any production angle of the motion mechanism, which is the generalized calibration matrix. It can be seen that this scheme can eliminate the influence of the difference between the calibration angle and the production angle on camera calibration.

[0183] Figure 7 This is a schematic diagram of the structure of the camera calibration device without angle constraints provided in the embodiments of this application, as shown below. Figure 7 As shown, the device may include the following modules:

[0184] The first construction module 710 is used to construct a translation calibration matrix corresponding to the camera based on the method of performing multiple specified parallel movements on the motion mechanism; wherein, the multiple specified parallel movements cause the calibration object to perform multiple parallel movements within the field of view of the camera.

[0185] The control module 720 is used to control the motion mechanism to rotate from a designated position to multiple auxiliary positions so that the calibration object rotates and moves multiple times relative to the camera, and to control the camera to take pictures when the motion mechanism moves to each auxiliary position to obtain an auxiliary image containing the calibration object.

[0186] The fitting module 730 is used to perform fitting processing on the pixel coordinates of each auxiliary image with respect to the calibration object about the rotation center, so as to obtain the pixel rotation center;

[0187] The second construction module 740 is used to construct a rotation matrix using the pixel rotation center and the angle parameters corresponding to the motion mechanism; wherein, the angle parameters represent the angle difference between the calibration angle of the motion mechanism and the production angle which is an unknown parameter, and the rotation matrix is ​​used to represent the transformation relationship between before and after the pixel point rotates around the pixel rotation center in the pixel coordinate system.

[0188] The correction module 750 is used to perform a specified correction process on the translation calibration matrix using the rotation matrix to obtain a generalized calibration matrix; wherein the specified correction process is used to ensure that the generalized calibration matrix is ​​used as the calibration result at the same calibration angle as the production angle at any production angle of the motion mechanism.

[0189] The device further includes:

[0190] An optimization module is used to normalize the generalized calibration matrix when a predetermined optimization condition is detected, to obtain an optimized generalized calibration matrix; wherein, the predetermined optimization condition is that the physical coordinates of the specified flange of the motion mechanism are different from the physical coordinates of the gripper of the motion mechanism, and the specified flange is a flange used to connect the gripper; the normalization process is used to convert the generalized calibration matrix into a matrix when the origin of the physical coordinate system corresponding to the motion mechanism is translated to the physical rotation center; the physical rotation center is a rotation center obtained by fitting the physical coordinates of each auxiliary position in the physical coordinate system.

[0191] When the camera is moving, the angle parameter is the target difference obtained by subtracting the production angle, which is an unknown parameter, from the calibrated angle of the motion mechanism.

[0192] When the camera is stationary, the angle parameter is the difference obtained by taking the negative of the target difference.

[0193] The second building module 740 may include:

[0194] The first determining submodule is used to determine a rotation formula that characterizes the transformation relationship between two pixels; wherein, the two pixels are the pixel before rotation and the pixel after rotation, and the pixel after rotation is obtained by rotating the pixel before rotation around the pixel rotation center by the angle difference represented by the angle parameter.

[0195] The submodule is used to obtain a target formula obtained by rewriting the rotation formula in a specified manner based on the rotation formula; wherein, the specified rewriting is: a homogeneous coordinate rewriting method that uses the pixels before rotation and a transformation matrix to represent the pixels after rotation.

[0196] The second determining submodule is used to determine the transformation matrix present in the target formula as a rotation matrix.

[0197] The rotation matrix is:

[0198]

[0199] Among them, M rot Let (α-β) be the rotation matrix, (α-β) represent the angle parameter, and CenterX and CenterY be the coordinate values ​​of the pixel rotation center.

[0200] The correction module 750 may include:

[0201] The correction submodule is used to substitute the rotation matrix and the translation calibration matrix into a predetermined correction formula to obtain a generalized calibration matrix; wherein, the predetermined correction formula includes: M推算 =M*(M rot ) -1 M 推算 Let M be the generalized calibration matrix, and M be the translation calibration matrix. rot ) -1 is the inverse of the rotation matrix.

[0202] The first building module 710 may include:

[0203] The control submodule is used to control the motion mechanism to perform multiple parallel movements so that the calibration object moves multiple times relative to the camera, and to control the camera to take pictures when the motion mechanism moves to each target position to obtain a target image containing the calibration object.

[0204] A generation submodule is used to generate a translation calibration matrix based on the pixel coordinates of the calibration object in each target image and the physical coordinates of the motion mechanism at each target position in the physical coordinate system.

[0205] Based on the above scheme, the translation calibration matrix can be used to characterize the transformation relationship between the pixel coordinate system and the physical coordinate system at the calibration angle of the motion mechanism, while the rotation matrix can be used to characterize the transformation relationship before and after a pixel rotates around its rotation center. Therefore, the rotation matrix can be used to perform specified correction processing on the translation calibration matrix to obtain a calibration result that can be used as the same calibration angle as the production angle at any production angle of the motion mechanism, which is the generalized calibration matrix. It can be seen that this scheme can eliminate the influence of the difference between the calibration angle and the production angle on camera calibration.

[0206] This application also provides an electronic device, such as... Figure 8 As shown, it includes:

[0207] Memory 801 is used to store computer programs;

[0208] When processor 802 executes a program stored in memory 801, it performs the following steps:

[0209] A translation calibration matrix corresponding to the camera is constructed by performing multiple specified parallel movements on the motion mechanism; wherein, the multiple specified parallel movements cause the calibration object to perform multiple parallel movements relative to the camera;

[0210] The motion mechanism is controlled to rotate from a designated position to multiple auxiliary positions so that the calibration object rotates and moves multiple times relative to the camera, and the camera is controlled to take pictures when the motion mechanism moves to each auxiliary position to obtain an auxiliary image containing the calibration object;

[0211] The pixel coordinates of the calibration object in each auxiliary image are fitted with respect to the rotation center to obtain the pixel rotation center.

[0212] A rotation matrix is ​​constructed using the pixel rotation center and the angle parameters corresponding to the motion mechanism; wherein, the angle parameters represent the angle difference between the calibration angle of the motion mechanism and the production angle which is an unknown parameter, and the rotation matrix is ​​used to represent the transformation relationship between the pixel point before and after rotation when the pixel point rotates around the pixel rotation center in the pixel coordinate system;

[0213] Using the rotation matrix, the translation calibration matrix is ​​modified to obtain a generalized calibration matrix; wherein, the specified modification is used to ensure that at any production angle of the motion mechanism, the generalized calibration matrix is ​​used as the calibration result at the same calibration angle as that production angle.

[0214] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 802, the communication interface, and the memory 801 communicating with each other via the communication bus.

[0215] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0216] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0217] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0218] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0219] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described camera calibration methods without angular constraints.

[0220] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the camera calibration methods without angular constraints described in the above embodiments.

[0221] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0222] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0223] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0224] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A camera calibration method without angle constraints, characterized in that, Applied to control equipment, the method includes: A translation calibration matrix corresponding to the camera is constructed by performing multiple specified parallel movements on the motion mechanism; wherein, the multiple specified parallel movements cause the calibration object to perform multiple parallel movements relative to the camera; The motion mechanism is controlled to rotate from a designated position to multiple auxiliary positions so that the calibration object rotates and moves multiple times relative to the camera, and the camera is controlled to take pictures when the motion mechanism moves to each auxiliary position to obtain an auxiliary image containing the calibration object; The pixel coordinates of the calibration object in each auxiliary image are fitted with respect to the rotation center to obtain the pixel rotation center. A rotation matrix is ​​constructed using the pixel rotation center and the angle parameters corresponding to the motion mechanism; wherein, the angle parameters represent the angle difference between the calibration angle of the motion mechanism and the production angle which is an unknown parameter, and the rotation matrix is ​​used to represent the transformation relationship between the pixel point before and after rotation when the pixel point rotates around the pixel rotation center in the pixel coordinate system; Using the rotation matrix, the translation calibration matrix is ​​modified to obtain a generalized calibration matrix; wherein, the specified modification is used to ensure that at any production angle of the motion mechanism, the generalized calibration matrix is ​​used as the calibration result at the same calibration angle as that production angle.

2. The method according to claim 1, characterized in that, The method further includes: If a predetermined optimization condition is detected, the generalized calibration matrix is ​​normalized to obtain an optimized generalized calibration matrix; wherein, the predetermined optimization condition is that the physical coordinates of the designated flange of the motion mechanism are different from the physical coordinates of the gripper of the motion mechanism, and the designated flange is the flange used to connect the gripper; the normalization process is used to convert the generalized calibration matrix into a matrix when the origin of the physical coordinate system corresponding to the motion mechanism is translated to the physical rotation center; the physical rotation center is the rotation center obtained by fitting the physical coordinates of each auxiliary position in the physical coordinate system.

3. The method according to claim 1, characterized in that, When the camera is moving, the angle parameter is the target difference obtained by subtracting the production angle, which is an unknown parameter, from the calibrated angle of the motion mechanism. When the camera is stationary, the angle parameter is the difference obtained by taking the negative of the target difference.

4. The method according to any one of claims 1-3, characterized in that, The step of constructing a rotation matrix using the pixel rotation center and the corresponding angle parameters of the motion mechanism includes: Determine a rotation formula that characterizes the transformation relationship between two pixels; wherein the two pixels are the pixel before rotation and the pixel after rotation, and the pixel after rotation is obtained by rotating the pixel before rotation around the pixel rotation center by the angle difference represented by the angle parameter; Based on the rotation formula, a target formula is obtained after the rotation formula is rewritten in a specified manner; wherein, the specified rewriting is: a homogeneous coordinate rewriting method that uses the pixel points before rotation and a transformation matrix to represent the pixel points after rotation. The transformation matrix present in the target formula is determined as a rotation matrix.

5. The method according to claim 4, characterized in that, The rotation matrix is: Among them, M rot Let (α-β) be the rotation matrix, (α-β) represent the angle parameter, and CenterX and CenterY be the coordinate values ​​of the pixel rotation center.

6. The method according to claim 5, characterized in that, The step of using the rotation matrix to perform a specified correction process on the translation calibration matrix to obtain a generalized calibration matrix includes: Substituting the rotation matrix and the translation calibration matrix into a predetermined correction formula yields the generalized calibration matrix; wherein the predetermined correction formula includes: M 推算 =M*(M rot ) -1 M 推算 Let M be the generalized calibration matrix, and M be the translation calibration matrix. rot ) -1 is the inverse of the rotation matrix.

7. The method according to claim 1, characterized in that, The method of constructing the translation calibration matrix corresponding to the camera based on multiple specified parallel movements of the motion mechanism includes: The motion mechanism is controlled to perform multiple parallel movements so that the calibration object moves multiple times relative to the camera, and the camera is controlled to take pictures when the motion mechanism moves to each target position to obtain a target image containing the calibration object. A translation calibration matrix is ​​generated based on the pixel coordinates of the calibration object in each target image and the physical coordinates of the motion mechanism at each target position in the physical coordinate system.

8. A camera calibration device without angle constraints, characterized in that, The device includes: The first construction module is used to construct the translation calibration matrix corresponding to the camera based on the method of performing multiple specified parallel movements on the motion mechanism; wherein, the multiple specified parallel movements cause the calibration object to perform multiple parallel movements relative to the camera; The control module is used to control the motion mechanism to rotate from a designated position to multiple auxiliary positions so that the calibration object rotates and moves multiple times relative to the camera, and to control the camera to take pictures when the motion mechanism moves to each auxiliary position to obtain an auxiliary image containing the calibration object; The fitting module is used to perform fitting processing on the pixel coordinates of each auxiliary image with respect to the calibration object about the rotation center, so as to obtain the pixel rotation center; The second construction module is used to construct a rotation matrix using the pixel rotation center and the angle parameters corresponding to the motion mechanism; wherein, the angle parameters represent the angle difference between the calibration angle of the motion mechanism and the production angle which is an unknown parameter, and the rotation matrix is ​​used to represent the transformation relationship between the pixel point before and after rotation when the pixel point rotates around the pixel rotation center in the pixel coordinate system; The correction module is used to perform a specified correction process on the translation calibration matrix using the rotation matrix to obtain a generalized calibration matrix; wherein the specified correction process is used to ensure that the generalized calibration matrix is ​​used as the calibration result at the same calibration angle as the production angle at any production angle of the motion mechanism.

9. The apparatus according to claim 8, characterized in that, The device further includes: An optimization module is used to normalize the generalized calibration matrix when a predetermined optimization condition is detected, to obtain an optimized generalized calibration matrix; wherein, the predetermined optimization condition is that the physical coordinates of the specified flange of the motion mechanism are different from the physical coordinates of the gripper of the motion mechanism, and the specified flange is a flange used to connect the gripper; the normalization process is used to convert the generalized calibration matrix into a matrix when the origin of the physical coordinate system corresponding to the motion mechanism is translated to the physical rotation center; the physical rotation center is a rotation center obtained by fitting the physical coordinates of each auxiliary position in the physical coordinate system.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images