A camera calibration method, device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a camera calibration method and device, electronic equipment and storage medium, and relate to the technical field of computer images. The method comprises: constructing a translation calibration matrix; controlling the end effector to rotate to a first auxiliary position where the calibration object is located outside the field of view of the camera; controlling the end effector to translate to a second auxiliary position where the calibration object is located within the field of view of the camera, and controlling the camera to take a picture at the second auxiliary position to obtain an auxiliary image; determining virtual coordinates of the end effector at the first auxiliary position and calibration object pixel coordinates by using pixel coordinates in the auxiliary image, first and second auxiliary position physical coordinates, and the translation calibration matrix; fitting a plurality of virtual coordinates and a plurality of first auxiliary position physical coordinates to obtain a mechanism rotation center; and determining a calibration result by using the mechanism rotation center and the translation calibration matrix. The scheme provided by the embodiments of the present application can improve the accuracy of camera calibration in a small field of view environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer image, in particular to a camera calibration method and device, electronic equipment and storage medium. BACKGROUND

[0002] In order to meet the needs of enterprise production, the related enterprises can apply the way of combining camera with motion mechanism to complete the production task. In the way of combining camera with motion mechanism, the related enterprises usually apply camera calibration technology for preprocessing. Through the calibration result obtained by the camera calibration process, the image of the camera can be used to guide the positioning of the motion mechanism in the enterprise production process.

[0003] In the related art, in the camera calibration process, the end effector of the motion mechanism needs to be rotated in addition to multiple specified parallel movements to make the calibration object move multiple times in parallel in the camera field of view. Based on the pixel coordinates of the calibration object obtained by the rotation movement and the physical coordinates of the end effector in the physical coordinate system, the center of the mechanism rotation is fitted and processed, and the center of the mechanism rotation is used to determine the calibration result.

[0004] However, when the ratio of the motion range of the calibration object in the camera field of view to the motion range of the product to be controlled in the camera field of view is small, it can be considered that the camera calibration process is in a small field of view environment. For example, when the motion range of the calibration object in the camera field of view is only one fifth of the motion range of the product to be controlled in the camera field of view, it can be considered that the camera calibration process is in a small field of view environment. At this time, since the motion range of the calibration object in the camera field of view is small, the calibration object can only rotate a small angle in the camera field of view. Therefore, when fitting the center of the mechanism rotation, it can be very sensitive to the small error of the physical coordinates or the pixel coordinates, so that the center of the mechanism rotation obtained by fitting is unstable, and the calibration result obtained by solving is large, that is, the accuracy of camera calibration is low.

[0005] Therefore, how to improve the accuracy of camera calibration in a small field of view environment is a problem to be solved. SUMMARY

[0006] The purpose of the embodiments of the present application is to provide a camera calibration method, device, electronic equipment and storage medium to improve the accuracy of camera calibration in a small field of view environment. The specific technical solutions are as follows:

[0007] In a first aspect, the embodiments of the present application provide a camera calibration method applied to a control device, the method comprising:

[0008] constructing a translation calibration matrix corresponding to the camera based on a plurality of specified parallel movements of an end effector of a motion mechanism; wherein the plurality of specified parallel movements cause the calibration object to move in parallel a plurality of times within a field of view of the camera;

[0009] controlling the end effector to move in rotation a plurality of times so that the end effector reaches a plurality of first auxiliary positions; wherein the first auxiliary positions are positions such that the calibration object is located outside the field of view of the camera;

[0010] controlling the end effector to move in parallel each time the first auxiliary position is reached so as to move the end effector to a second auxiliary position, and controlling the camera to take a photograph each time the end effector moves to each second auxiliary position, to obtain an auxiliary image containing the calibration object; wherein the second auxiliary positions are positions such that the calibration object is located within the field of view of the camera;

[0011] determining a plurality of virtual coordinates using the pixel coordinates of the calibration object in each auxiliary image, the physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and the translation calibration matrix; wherein the virtual coordinates are pixel coordinates of the calibration object in a camera coordinate system when the end effector is located at the first auxiliary positions;

[0012] performing fitting processing on the plurality of virtual coordinates and the physical coordinates of the end effector at each first auxiliary position with respect to a rotation center, to obtain a mechanism rotation center;

[0013] determining a calibration result corresponding to the camera using the mechanism rotation center and the translation calibration matrix.

[0014] Optionally, the determining a plurality of virtual coordinates using the pixel coordinates of the calibration object in each auxiliary image, the physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and the translation calibration matrix comprises:

[0015] determining a plurality of virtual coordinates using the pixel coordinates of the calibration object in each auxiliary image, the physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and an intermediate matrix; wherein, in the case of a stationary camera, the intermediate matrix is the translation calibration matrix, and in the case of a moving camera, the intermediate matrix is a generalized calibration matrix obtained by performing correction processing on the translation calibration matrix, the correction processing being used to convert the translation calibration matrix into a generalized calibration matrix representing the transformation relationship between a rotated camera coordinate system and a physical coordinate system, the rotated camera coordinate system being a camera coordinate system formed after the camera follows the motion mechanism to move in rotation.

[0016] Optionally, the mechanism rotation center comprises: a pixel rotation center in a camera coordinate system and a physical rotation center in a physical coordinate system.

[0017] The method further comprises:

[0018] The method further comprises:

[0019] Optionally, the method of determining the plurality of virtual coordinates comprises:

[0020] The method further comprises:

[0021] Optionally, the translation transformation equation is an equation based on a plane Euclidean transformation principle and used to represent a transformation relationship between the pixel coordinates of the calibration object in the auxiliary images, the physical coordinates of the end effector at the first auxiliary position and the second auxiliary position, the intermediate matrix and the virtual coordinates.

[0022] Optionally, the translation transformation equation in the case of a static camera is M*P′-M*P=W1-W2, and the translation transformation equation in the case of a moving camera is M t *P′-M t *P=W2-W1.

[0023] Optionally, M is a translation calibration matrix as the intermediate matrix, P is the pixel coordinates of the calibration object in the auxiliary images, P′ is the virtual coordinates, W1 is the physical coordinates of the end effector at the first auxiliary position, W2 is the physical coordinates of the end effector at the second auxiliary position, and M x is a generalized calibration matrix as the intermediate matrix.

[0024] Optionally, the method of correcting the translation calibration matrix comprises:

[0025] The translation calibration matrix is corrected by using a rotation angle of the end effector in the rotation movement and a projective transformation equation to obtain a generalized calibration matrix; the projective transformation equation is an equation obtained based on a projective transformation principle and used to represent a transformation relationship among the rotation angle of the end effector in the rotation movement, the translation calibration matrix and the generalized calibration matrix.

[0026] Optionally, the projective transformation equation is: wherein θ represents the rotation angle of the end effector in the rotation movement, T x and T y is a displacement amount of the end effector in parallel movement from the first auxiliary position to the second auxiliary position, M t is a generalized calibration matrix as the intermediate matrix, and M is a translation calibration matrix as the intermediate matrix.

[0027] Optionally, the normalization processing of the translation calibration matrix by using the pixel rotation center includes:

[0028] The translation component in the translation calibration matrix is represented by the pixel rotation center and the rotation angle of the end effector in the rotation movement by using an equation M*C=0; wherein C is a coordinate of the pixel rotation center, and M is the translation calibration matrix.

[0029] Optionally, the translation calibration matrix corresponding to the camera is constructed based on the multiple specified parallel movements of the end effector of the motion mechanism, and the multiple specified parallel movements make the calibration object move in the field of view of the camera multiple times.

[0030] The end effector of the motion mechanism is controlled to move in multiple specified parallel movements so that the calibration object moves in the field of view of the camera multiple times.

[0031] The camera is controlled to take pictures when the end effector moves to each target position to obtain target images containing the calibration object.

[0032] The translation calibration matrix is generated based on pixel coordinates of the calibration object in each target image and physical coordinates of the end effector in each target position in the physical coordinate system.

[0033] In a second aspect, an embodiment of the present application provides a camera calibration device, and the device includes:

[0034] A construction module is configured to construct a translation calibration matrix corresponding to the camera based on multiple specified parallel movements of the end effector of the motion mechanism; the multiple specified parallel movements make the calibration object move in the field of view of the camera multiple times.

[0035] The first control module is configured to control the end effector to perform multiple rotational movements so that the end effector reaches multiple first auxiliary positions; wherein the first auxiliary position is a position in which the calibration object is located outside the field of view of the camera.

[0036] The second control module is configured to control the end effector to perform a parallel movement to move the end effector to a second auxiliary position when the end effector reaches each first auxiliary position, and control the camera to take a picture when the end effector moves to each second auxiliary position, to obtain an auxiliary image containing the calibration object; wherein the second auxiliary position is a position in which the calibration object is located inside the field of view of the camera.

[0037] The first determination module is configured to determine multiple virtual coordinates by using pixel coordinates of the calibration object in each auxiliary image, physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and the translation calibration matrix; wherein the virtual coordinate is a pixel coordinate of the calibration object in the camera coordinate system when the end effector is located at the first auxiliary position.

[0038] The fitting module is configured to perform fitting processing on the multiple virtual coordinates and the physical coordinates of the end effector at each first auxiliary position with respect to the rotation center, to obtain the mechanism rotation center.

[0039] The second determination module is configured to determine the calibration result corresponding to the camera by using the mechanism rotation center and the translation calibration matrix.

[0040] Embodiments of the present application further provide an electronic device, which comprises:

[0041] The memory is configured to store a computer program.

[0042] The processor is configured to execute the program stored on the memory, to implement the camera calibration method.

[0043] Embodiments of the present application further provide a computer readable storage medium, which stores a computer program; the computer program is executed by a processor to implement the camera calibration method.

[0044] Embodiments of the present application have the following beneficial effects:

[0045] The method for camera calibration provided in the embodiments of the present application can construct a translation calibration matrix corresponding to the camera based on multiple specified parallel movements of the end effector of the motion mechanism; the multiple specified parallel movements can cause the calibration object to move in parallel multiple times in the field of view of the camera; then the end effector is controlled to move in rotation multiple times so that the end effector reaches multiple first auxiliary positions at which the calibration object is located outside the field of view of the camera; when reaching each first auxiliary position, the end effector is controlled to move in parallel so as to move the end effector to a second auxiliary position at which the calibration object is located in the field of view of the camera, and the camera is controlled to take a picture when the end effector moves to each second auxiliary position, thereby obtaining an auxiliary image containing the calibration object; then the pixel coordinates of the calibration object in each auxiliary image, the physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and the aforementioned constructed translation calibration matrix are used to determine multiple virtual coordinates; the virtual coordinates are the pixel coordinates of the calibration object in the camera coordinate system when the end effector is located at the first auxiliary position; then the multiple virtual coordinates and the physical coordinates of the multiple first auxiliary positions are subjected to fitting processing with respect to the rotation center, thereby obtaining the mechanism rotation center; the aforementioned obtained mechanism rotation center and the constructed translation calibration matrix are used to determine the calibration result corresponding to the camera.

[0046] Based on the above scheme, after the end effector moves in rotation, it reaches multiple first auxiliary positions at which the calibration object is located outside the field of view of the camera, and the pixel coordinates of the calibration object in the camera coordinate system when the end effector is located at each first auxiliary position, i.e., the virtual coordinates, are determined; then the multiple virtual coordinates and the physical coordinates of the end effector at each first auxiliary position are subjected to fitting processing with respect to the rotation center, thereby obtaining the mechanism rotation center; in this way, in the process of fitting the mechanism rotation center, the pixel coordinates of the calibration object can include the pixel coordinates outside the field of view of the camera, and the physical coordinates of the end effector can include the physical coordinates of the first auxiliary positions, which is equivalent to expanding the field of view of the camera. Therefore, by fitting the pixel coordinates and the physical coordinates according to the present scheme, the sensitivity to the error of the physical coordinates or the pixel coordinates can be reduced, the stability of the obtained mechanism rotation center is improved, the error of the calibration result is reduced, and the precision of the camera calibration is improved.

[0047] Of course, implementing any product or method of the present application does not necessarily require achieving all the advantages described above at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and other embodiments can also be obtained by those skilled in the art based on these drawings.

[0049] Figure 1 The principle diagram of error amplification of the traditional calibration method fitting the rotation center of the mechanism in the case of disturbance in a small field of view environment;

[0050] Figure 2 The structure diagram of the calibration control system of the camera in a static state provided by the embodiments of the present application;

[0051] Figure 3 The flow diagram of a camera calibration method provided by the embodiments of the present application;

[0052] Figure 4 The flow diagram of a camera calibration method provided by the embodiments of the present application;

[0053] Figure 5 The flow diagram of a camera calibration method provided by the embodiments of the present application;

[0054] Figure 6 The flow diagram of a camera calibration method provided by the embodiments of the present application;

[0055] Figure 7 The structure diagram of a camera calibration device provided by the embodiments of the present application;

[0056] Figure 8 The structure diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art based on the present application belong to the scope of protection of the present application.

[0058] In order to facilitate the understanding of the scheme, first, the professional terms involved in the embodiments of the present application are introduced.

[0059] Euclidean transformation: the shape and size of an object remain unchanged after a composite transformation of translation and rotation, such composite transformation is called Euclidean transformation; wherein, the object whose shape and size remain unchanged after the composite transformation of translation and rotation can be called a rigid body.

[0060] Similarity transformation: The shape of an object remains unchanged after a composite transformation of a Euclidean transformation and a uniform scaling transformation, but the size of the object changes. Such a composite transformation is called a similarity transformation.

[0061] Affine transformation: The shape of an object does not remain unchanged after a composite transformation of a non-singular linear transformation and a translation transformation, but parallel lines remain parallel. Such a composite transformation is called an affine transformation.

[0062] Projective transformation: The shape of an object remains unchanged after a projection transformation between any two planes, but the cross ratio of collinear points remains unchanged. Such a transformation is called a projective transformation. The projection transformation between any two planes is also called a homography transformation.

[0063] Homogeneous linear transformation: A linear transformation between homogeneous vectors is called a homogeneous linear transformation. A homogeneous vector can be a vector that only differs from another vector by a non-zero global scaling factor.

[0064] Calibration: The process of establishing a geometric model of camera imaging to determine the geometric relationship between a point on the surface of a space object and its corresponding point in an image. The parameters of these geometric models can be camera parameters. Under most conditions, these parameters must be obtained through experiments and calculations. This process of solving parameters is called camera calibration. The purpose of camera calibration is to solve the projective transformation matrix representing the transformation relationship between the camera coordinate system and the physical coordinate system belonging to the motion mechanism.

[0065] Calibration matrix: The result of calibration is a calibration matrix.

[0066] Calibration object: An intermediary that can be used to complete calibration, such as a calibration board, materials, etc.

[0067] Field of view (FOV, Field angle Of View): The actual range of objects that can be accommodated within the field of view seen by the camera.

[0068] Disturbance: During the calibration process, system errors or noise will exist due to the positioning accuracy of the motion mechanism, the image imaging quality, and the positioning accuracy of the feature point extraction. The deviation introduced by this is called disturbance deviation.

[0069] Eye-in-hand: Eye-in-hand refers to a camera mounted on a motion mechanism and moving with the motion mechanism. Therefore, eye-in-hand can also be called camera movement.

[0070] Eye-to-hand: Eye-to-hand refers to a camera mounted outside the motion mechanism and kept in a stationary state. Therefore, eye-to-hand can also be called camera stationary.

[0071] Rotation center: the center of rotation axis when the motion mechanism rotates.

[0072] In order to better understand the embodiments of the present application, the prior art will be introduced below.

[0073] In the related art, during the movement of the motion mechanism, the calibration object can be moved in parallel and rotated multiple times in the camera field of view, and at the same time, the camera collects multiple images containing the calibration object, and obtains multiple matched physical point sets and image point sets, wherein the physical point set can include multiple physical coordinates of the end effector belonging to the motion mechanism in the physical coordinate system, and the image point set can include multiple pixel coordinates of the calibration object in the collected image; then, based on the multiple sets of physical coordinates and pixel coordinates, a nonlinear optimization method is used to obtain the corresponding relationship between the camera coordinate system and the physical coordinate system that minimizes the re-projection error, thereby obtaining an initial homography matrix; then, the multiple sets of physical coordinates and pixel coordinates are fitted to obtain the mechanism rotation center, and the initial homography matrix is normalized based on the mechanism rotation center, thereby obtaining a calibration matrix, i.e., the calibration result.

[0074] However, in a small field of view environment, the movement range of the calibration object in the camera field of view is small, so the calibration object can only rotate a small angle in the camera field of view, and thus when fitting the mechanism rotation center, it can be very sensitive to the very small error of the physical coordinates or pixel coordinates, thereby causing the fitted mechanism rotation center to be unstable, and the obtained calibration result to have a large error, i.e., the accuracy of camera calibration is low.

[0075] Exemplary, Figure 1 In the case of disturbances in a small field of view environment, the principle of error amplification of the traditional calibration method for fitting the mechanism rotation center is shown in the schematic diagram. As Figure 1 shown, A=(1420.09, 988.88), B=(1115.86, 973.64), C=(803.56, 985.45), D=(1420.29, 988.67), E=(1115.66, 973.14), F=(803.76, 985.35), c:(x-1092.35) 2 +(y-4487.12) 2 =12345126.89, d:(x-1093.64) 2 +(y-4400.52) 2= 11747436.54. A, B, C are the pixel coordinates of three groups of calibration objects sampled, and the circle c is fitted by A, B, and C. At this time, a disturbance error is artificially applied, and 0.2 pixels of noise is applied to the pixel coordinates of the three groups of calibration objects A, B, and C. The pixel coordinates of the three groups of calibration objects obtained are D, E, and F. The circle d is fitted by D, E, and F. Since the coordinates of A, B, C and D, E, F are very small, the circles are basically coincident in the figure, and A, B, and C are blocked by D, E, and F. From the equations of the circle c and the circle d, when the pixel coordinates of the fitting points change by 0.2 pixels due to noise interference, the center coordinates X of the fitted circles differ by 1.29 pixels, the center coordinates Y of the fitted circles differ by 86.6 pixels, and the error amplification factor is about 86.6 ÷ 0.2 = 433 times.

[0076] It can be seen that how to improve the accuracy of camera calibration in a small field of view environment is a problem to be solved.

[0077] In order to improve the accuracy of camera calibration in a small field of view environment, the embodiment of the application provides a camera calibration method, device, electronic equipment and storage medium.

[0078] First, a camera calibration method provided by the embodiment of the application is introduced.

[0079] The camera calibration method provided by the embodiment of the application can be applied to a control device. In the specific application process, the control device, the camera and the motion mechanism can constitute a calibration control system. The camera can be used for photographing the calibration object and sending the image containing the calibration object to the control device. The motion mechanism can move in parallel and rotate to move the end effector to a specified position. The control device can control the motion of the motion mechanism and the camera, receive the image sent by the camera, obtain the position of the end effector, and determine the calibration result based on the obtained content.

[0080] In order to better understand the camera calibration system, the camera calibration system is described in conjunction with Figure 2 Exemplary description: for the case where the camera is stationary, such as Figure 2As shown, the camera 1 is fixed at a position outside the motion mechanism, and the suction nozzle 4 of the motion mechanism can be connected with the calibration object 2, that is, the suction nozzle 2 sucks the calibration object 2, and the suction nozzle 4 is connected with the flange 3. In a specific application, in a default physical coordinate system of the motion mechanism, the physical coordinates of the flange 3 are usually taken as the physical coordinates of the end effector; in another case, the motion mechanism can set a tool (Tool Center Point, TCP) coordinate system, and the physical coordinates of the suction nozzle 4 are taken as the physical coordinates of the end effector; in the above two cases, the physical coordinates of the end effector can be obtained; specifically, the camera 1 can be used to take a picture of the calibration object 2 and send an image containing the calibration object 2 to the control device; the control device can control the flange 3 and the suction nozzle 4 to move, thereby driving the calibration object 2 to move; the control device can also control the camera 1 to take a picture, receive the image sent by the camera 1, obtain the physical coordinates of the end effector in the physical coordinate system, and determine the calibration result based on the obtained content.

[0081] In the embodiments of the present application, a camera calibration method is provided, which can include the following steps:

[0082] A translation calibration matrix corresponding to the camera is constructed based on a plurality of specified parallel movements of an end effector of the motion mechanism; wherein the plurality of specified parallel movements cause the calibration object to move in the field of view of the camera multiple times.

[0083] The end effector is controlled to move multiple times in rotation so that the end effector reaches a plurality of first auxiliary positions; wherein the first auxiliary position is a position such that the calibration object is located outside the field of view of the camera.

[0084] When reaching each first auxiliary position, the end effector is controlled to move in parallel to move the end effector to a second auxiliary position, and the camera is controlled to take a picture when the end effector moves to each second auxiliary position, thereby obtaining an auxiliary image containing the calibration object; wherein the second auxiliary position is a position such that the calibration object is located within the field of view of the camera.

[0085] A plurality of virtual coordinates are determined using the pixel coordinates of the calibration object in each auxiliary image, the physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and the translation calibration matrix; wherein the virtual coordinates are the pixel coordinates of the calibration object in the camera coordinate system when the end effector is located at the first auxiliary position.

[0086] The plurality of virtual coordinates and the physical coordinates of the end effector at each first auxiliary position are fitted with respect to the rotation center, thereby obtaining the mechanism rotation center.

[0087] The rotation center of the mechanism and the translation calibration matrix are used to determine a calibration result corresponding to the camera.

[0088] According to the above scheme, after the end effector moves by rotation, the end effector reaches a plurality of first auxiliary positions at which the calibration object is located outside the field of view of the camera, and the pixel coordinates of the calibration object in the camera coordinate system, i.e., virtual coordinates, are determined when the end effector is located at each first auxiliary position. Then, the plurality of virtual coordinates and the physical coordinates of the end effector at the respective first auxiliary positions are fitted with respect to the rotation center to obtain the rotation center of the mechanism. In this way, in the process of fitting the rotation center of the mechanism, the pixel coordinates of the calibration object can include the pixel coordinates outside the field of view of the camera, and the physical coordinates of the end effector can include the physical coordinates of the first auxiliary positions, which is equivalent to expanding the field of view of the camera. Therefore, by fitting the pixel coordinates and the physical coordinates according to the scheme, the sensitivity to errors of the physical coordinates or the pixel coordinates can be reduced, the stability of the obtained rotation center of the mechanism is improved, the error of the calibration result is reduced, and the accuracy of camera calibration is improved.

[0089] A camera calibration method provided by an embodiment of the present application will be described below with reference to the accompanying drawings.

[0090] Figure 3 A flowchart of a camera calibration method provided by an embodiment of the present application is shown in FIG. 1, which can include steps S301-S306. Figure 3

[0091] S301, based on a plurality of specified parallel movements of an end effector of a motion mechanism, a translation calibration matrix corresponding to a camera is constructed; wherein the plurality of specified parallel movements enable a calibration object to move in parallel in the field of view of the camera.

[0092] It can be understood that the control device can control the end effector of the motion mechanism to move in parallel for a plurality of times while ensuring that the calibration object is in the field of view of the camera. At this time, the calibration object can also move in parallel in the field of view of the camera for a plurality of times. After each parallel movement, the control device can obtain a plurality of physical coordinates of the end effector and a plurality of pixel coordinates of the calibration object. Through the correspondence between the physical coordinates and the pixel coordinates, a matrix corresponding to the camera and used to represent the transformation relationship between the camera coordinate system and the physical coordinate system, i.e., a translation calibration matrix, can be constructed.

[0093] The calibration object can be an intermediary for determining the conversion relationship between the pixel coordinates and the physical coordinates, for example, a checkerboard calibration board, a two-dimensional code calibration board, a material, etc. The specific form of the calibration object is not limited in the present application.

[0094] ​In an implementation, the way of constructing the translation calibration matrix of the camera based on the multiple specified parallel movements of the end effector of the motion mechanism can include steps A1-A3:

[0095] A1, controlling the end effector of the motion mechanism to perform multiple specified parallel movements so that the calibration object moves multiple times in parallel in the field of view of the camera.

[0096] It can be understood that the control device can send a control signal to the motion mechanism to control the end effector of the motion mechanism to perform multiple specified parallel movements; wherein the multiple specified parallel movements can be at least four or more parallel movements of the end effector, and any three positions after movement are not collinear, and the calibration object can be ensured to be in the field of view of the camera after each parallel movement.

[0097] It should be noted that under the control of the control device, the calibration object can move multiple times in parallel in the field of view of the camera, and each parallel movement can be a displacement according to a fixed distance, and of course the distance of each parallel movement can also be different, which is also reasonable. The specific parallel movement form of the calibration object is not limited by the embodiments of the application.

[0098] A2, controlling the camera to take a picture when the end effector moves to each target position, to obtain a target image containing the calibration object.

[0099] It can be understood that the control device can control the camera to take a picture of the calibration object after each parallel movement to a target position, and obtain the image taken by the camera.

[0100] Wherein, each target position can be the position of the end effector in the three-dimensional world, and the specific form can be a three-dimensional coordinate, however, during the calibration process, the end effector only performs parallel movement, that is, the end effector in a certain dimension in the three-dimensional world can be fixed, therefore, the physical coordinates of the end effector can be two-dimensional coordinates.

[0101] It should be noted that each time the end effector moves to a target position, the calibration object can be in the field of view of the camera, and move a fixed distance to reach the corresponding target position; wherein the fixed distance can be a displacement in the same plane in the front, back, left, and right directions, for example, in the (x, y) two-dimensional coordinate system, a certain calibration object is moved in the positive direction of the x-axis by a unit distance, which can be referred to as a fixed distance. Therefore, the target image containing the calibration object obtained can include the pixel coordinates of the calibration object in the camera coordinate system; in addition, the pixel coordinates from the target image can be the coordinates of a point of a two-dimensional image, therefore, the pixel coordinates of the calibration object can be two-dimensional coordinates.

[0102] For the determination of the pixel coordinates of the calibration object, a feature point existing in the physical space can be selected as an anchor point on the calibration object to locate the calibration object in the image. After each target image is acquired, the feature point in the image can be determined as an image point in each target image containing the feature point of the calibration object, and the coordinates of the image point can be taken as the pixel coordinates of the calibration object. Meanwhile, in order to obtain complete and accurate pixel coordinates, the selected image point can be a point in the image with clear pixels and determined coordinate values.

[0103] A3, generating a translation calibration matrix based on the pixel coordinates of the calibration object in each target image and the physical coordinates of the end effector at each target position in the physical coordinate system.

[0104] It can be understood that the control device can control the end effector to move by a known displacement, and thus the control device can acquire the physical coordinates of the end effector in the physical coordinate system. Through the correspondence between the physical coordinates of the end effector and the pixel coordinates in the target image, a matrix corresponding to the camera and used to represent the transformation relationship between the camera coordinate system and the physical coordinate system, that is, the translation calibration matrix, can be constructed.

[0105] Specifically, the generation of the translation calibration matrix can be based on the principle of affine transformation to find the transformation relationship between the pixel coordinates of the calibration object and the physical coordinates of the end effector, so as to determine the transformation relationship between the pixel coordinate system in which the pixel coordinates are located and the physical coordinate system in which the physical coordinates are located. The transformation relationship can be in the form of a matrix, that is, the translation transformation matrix. The application does not limit the generation mode of the translation calibration matrix, and any generation mode of the translation calibration matrix can be applied to the application.

[0106] S302, controlling the end effector to move by rotation multiple times so that the end effector reaches multiple first auxiliary positions; wherein the first auxiliary position is a position in which the calibration object is located outside the field of view of the camera.

[0107] It can be understood that the end effector of the motion mechanism can be controlled by the control device to move by rotation multiple times within the movement range of the end effector. After the movement by rotation, the end effector can reach the first auxiliary position, and the calibration object can reach a position outside the field of view of the camera. At the position outside the field of view of the camera, the camera cannot take a picture of the calibration object, and thus when the end effector is located at the first auxiliary position, the physical coordinates of the end effector can be acquired, but the pixel coordinates of the calibration object cannot be directly acquired. The rotation angle of the end effector when moving by rotation can be a reasonable value within the movement range of the motion mechanism.

[0108] It should be noted that, in a process of one rotation movement, the rotation direction of the end effector can be clockwise rotation or counterclockwise rotation, and in the next rotation movement, it can be rotation movement in the opposite direction of the last time, or it can be rotation movement in the same direction of the last time, which is not limited in the embodiments of the present application.

[0109] S303, in each time the first auxiliary position is reached, the end effector is controlled to move in parallel to move the end effector to a second auxiliary position, and the camera is controlled to take a picture when the end effector moves to each second auxiliary position, to obtain an auxiliary image containing the calibration object; wherein the second auxiliary position is a position such that the calibration object is located in the field of view of the camera.

[0110] It can be understood that the control device can control the end effector to move in parallel from the first auxiliary position to the second auxiliary position, and the control device can control the camera to take a picture each time the end effector reaches the second auxiliary position; since the end effector is located at the second auxiliary position, the calibration object is in the field of view of the camera, at this time, the camera takes a picture, and the auxiliary image containing the calibration object can be obtained. The feature points of the calibration object in the auxiliary image can be used as image points, and the pixel coordinates of the calibration object in the auxiliary image can be obtained.

[0111] S304, using the pixel coordinates of the calibration object in each auxiliary image, the physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and the translation calibration matrix, a plurality of virtual coordinates are determined; wherein the virtual coordinates are the pixel coordinates of the calibration object in the camera coordinate system when the end effector is located at the first auxiliary position.

[0112] It can be understood that in the foregoing steps, the pixel coordinates of the calibration object in each auxiliary image, the physical coordinates of the end effector at the first auxiliary position and the second auxiliary position, and the translation calibration matrix can be obtained; and the displacement from the first auxiliary position to the second auxiliary position can be directly obtained by the physical coordinates of the end effector at the first auxiliary position and the second auxiliary position, or the physical coordinates of the end effector can be obtained by converting the pixel coordinates of the calibration object using the translation calibration matrix; therefore, when the pixel coordinates of the calibration object in each auxiliary image, the physical coordinates of the end effector at the first auxiliary position and the second auxiliary position, and the translation calibration matrix are known, the pixel coordinates of the calibration object when the end effector is located at the first auxiliary position, that is, the virtual coordinates of the calibration object located outside the field of view, can be obtained.

[0113] S305, fitting processing is performed on the plurality of virtual coordinates and the physical coordinates of the end effector at each first auxiliary position with respect to the rotation center, to obtain the mechanism rotation center.

[0114] It can be understood that, in the fitting process of the rotation center, a circle can be fitted by a plurality of point coordinates, and the center of the circle is the rotation center; if the point coordinates subjected to the fitting process are a plurality of virtual coordinates and a plurality of physical coordinates of the first auxiliary positions, the center of the fitted circle can be the mechanism rotation center.

[0115] S306, determining a calibration result corresponding to the camera by using the mechanism rotation center and the translation calibration matrix.

[0116] It can be understood that, the translation calibration matrix can be converted into a target matrix for a field of view of the calibrated camera by using the mechanism rotation center, the target matrix can represent a transformation relationship between the camera coordinate system and the physical coordinate system, and the target matrix can be used as the calibration result.

[0117] In an implementation manner, the mechanism rotation center includes a pixel rotation center in the camera coordinate system and a physical rotation center in the physical coordinate system.

[0118] It can be understood that, if the point coordinates subjected to the fitting process are a plurality of virtual coordinates, the center of the fitted circle can be the pixel rotation center; if the point coordinates subjected to the fitting process are a plurality of physical coordinates of the first auxiliary positions, the center of the fitted circle can be the physical rotation center.

[0119] The physical rotation center can be used as part of the calibration result, and the motion mechanism is positioned and navigated in the production process.

[0120] In addition, the step of determining the calibration result corresponding to the camera by using the mechanism rotation center and the translation calibration matrix can include the following steps:

[0121] normalizing the translation calibration matrix by using the pixel rotation center to obtain the calibration result corresponding to the camera; wherein the normalization is used to convert the translation calibration matrix into a matrix when the origin of the physical coordinate system corresponding to the motion mechanism is translated to the physical rotation center.

[0122] It can be understood that, the coordinates of the pixel rotation center can be brought into the translation calibration matrix by the normalization, and the obtained matrix can be used as the calibration result for the field of view of the calibrated camera.

[0123] Specifically, in an implementation manner, the step of normalizing the translation calibration matrix by using the pixel rotation center to obtain the calibration result corresponding to the camera can include the following steps:

[0124] The translation component in the translation calibration matrix is characterized by the coordinates of the pixel rotation center and a rotation angle of the rotational movement of the end effector, by using the equation M*C=0; wherein C is the coordinates of the pixel rotation center, and M is the translation calibration matrix.

[0125] It can be understood that the value of the displacement of the parallel movement of the end effector to the target position in the translation calibration process, that is, the translation component in the translation calibration matrix, can be expressed by the coordinates of the pixel rotation center and the rotation angle of the rotational movement of the end effector, to obtain a target matrix; wherein the target matrix can be used as a calibration result of the field of view of the calibrated camera, and can be used to represent the transformation relationship between the camera coordinate system and the physical coordinate system.

[0126] For example, if the translation calibration matrix is Pixel rotation center Wherein, θ represents the rotation angle of the rotational movement of the end effector, and a and b represent the value of the displacement of the parallel movement of the end effector to the target position in the translation calibration process, in the process of normalizing the translation calibration matrix, the following equation can be determined by using the equation M*C=0:

[0127]

[0128] Therefore, the following equation can be determined:

[0129] a+(CenterX*cosθ-CenterY*sinθ)=0,b+(CenterX*sinθ+CenterY*cosθ)=0;

[0130] Therefore, the translation component in the M matrix can be determined as:

[0131] a=-(CenterX*cosθ-CenterY*sinθ),b=-(CenterX*sinθ+CenterY*cosθ);

[0132] The parametric expression of a and b is brought into M to obtain a target matrix:

[0133]

[0134] It should be noted that the translation and rotation transformation in the 2D projection space belong to homogeneous linear transformation, and the matrix processing involved in the embodiment includes but is not limited to: construction of the translation calibration matrix, acquisition of the generalized calibration matrix, and acquisition of the target calibration matrix, which can all be processing about homogeneous coordinate operation.

[0135] Based on the above scheme, after the end effector moves by rotation, it reaches a plurality of first auxiliary positions such that the calibration object is located outside the field of view of the camera, and the pixel coordinates of the calibration object in the camera coordinate system, i.e. virtual coordinates, are determined when the end effector is located at each first auxiliary position. Then, the plurality of virtual coordinates and the physical coordinates of the end effector at each first auxiliary position are subjected to fitting processing with respect to the rotation center to obtain the mechanism rotation center. In this way, in the process of fitting the mechanism rotation center, the pixel coordinates of the calibration object can include the pixel coordinates outside the field of view of the camera, and the physical coordinates of the end effector can include the physical coordinates of the first auxiliary positions, which is equivalent to expanding the field of view of the camera. Therefore, by fitting the pixel coordinates and the physical coordinates according to the present scheme, the sensitivity to errors of the physical coordinates or the pixel coordinates can be reduced, the stability of the obtained mechanism rotation center is improved, the error of the calibration result is reduced, and the accuracy of the camera calibration is improved.

[0136] In addition, in the present embodiment, the range of the pixel coordinates of the calibration object can be expanded by obtaining the virtual coordinates outside the field of view of the camera, which is equivalent to expanding the field of view of the camera, avoiding the influence of disturbances in a small field of view environment, solving the problem of magnification of calibration errors in a small field of view disturbance environment, and realizing high-precision camera calibration in a small field of view environment.

[0137] In addition, in the present embodiment, the physical coordinates of the motion mechanism and the pixel coordinates of the calibration object positioned using the feature points of the calibration object can be obtained by controlling the device to achieve the goal of camera calibration, without relying on higher-precision motion mechanisms or calibration boards, and the cost of high-precision camera calibration is significantly reduced.

[0138] In addition, in the present embodiment, a small number of steps can be added to the conventional camera calibration process to obtain various coordinates corresponding to a larger fitting circle, and the operation process is simple, while the algorithm complexity is almost unchanged, and the robustness of the calibration is significantly improved.

[0139] In addition, in the present embodiment, two-dimensional pixel coordinates and physical coordinates can be used in the process of constructing the translation calibration matrix and the target matrix, so that camera calibration of various two-dimensional scenes can be compatible, and the portability is higher.

[0140] In addition, in the present embodiment, the translation calibration matrix can be normalized using the mechanism rotation center to reduce the complexity of subsequent positioning operations and improve the practicality of the calibration result.

[0141] Optionally, in another embodiment, based on the camera calibration method shown in Figure 3 Based on the camera calibration method shown in the above embodiment, the determination of the plurality of virtual coordinates using the pixel coordinates of the calibration object in each auxiliary image, the physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and the translation calibration matrix can include the following steps:

[0142] determining a plurality of virtual coordinates by using the pixel coordinates of the calibration object in each auxiliary image, the physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and an intermediate matrix; wherein, in the case of a stationary camera, the intermediate matrix is the translation calibration matrix, and in the case of a moving camera, the intermediate matrix is a generalized calibration matrix obtained by modifying the translation calibration matrix, the modification being used to convert the translation calibration matrix into a generalized calibration matrix representing the transformation relationship between the rotated camera coordinate system and the physical coordinate system, the rotated camera coordinate system being a camera coordinate system formed after the camera follows the motion mechanism to move rotationally.

[0143] It can be understood that, in the case of a stationary camera, the plurality of virtual coordinates can be determined directly by using the pixel coordinates of the calibration object in each auxiliary image, the physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and the translation calibration matrix obtained in the foregoing step; while in the case of a moving camera, the plurality of virtual coordinates can be determined by using the pixel coordinates of the calibration object in each auxiliary image, the physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and the generalized calibration matrix obtained by modifying the translation calibration matrix.

[0144] For the intermediate matrix, in one camera calibration process, there can be an intermediate matrix as the intermediate matrix; in the case of a stationary camera, the intermediate matrix can be the translation calibration matrix; while in the case of a moving camera, the intermediate matrix can be a generalized calibration matrix obtained by modifying the translation calibration matrix.

[0145] For the modification, the rotation angle of the camera following the motion mechanism to move rotationally can be introduced into the translation calibration matrix to obtain a generalized calibration matrix representing the transformation relationship between the rotated camera coordinate system and the physical coordinate system; wherein, the rotation angle can be the rotation angle of the end effector to move rotationally.

[0146] In an implementation manner, the determining a plurality of virtual coordinates by using the pixel coordinates of the calibration object in each auxiliary image, the physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and an intermediate matrix can include the following steps:

[0147] The virtual coordinates are determined according to a translation transformation equation, pixel coordinates of the calibration object in each auxiliary image, physical coordinates of the end effector at the first auxiliary position and the second auxiliary position, and the intermediate matrix; wherein the translation transformation equation is an equation obtained based on a principle of a planar Euclidean transformation and used to represent a transformation relationship between the pixel coordinates of the calibration object in the auxiliary image, the physical coordinates of the end effector at the first auxiliary position and the second auxiliary position, the intermediate matrix, and the virtual coordinates.

[0148] It can be understood that the displacement from the first auxiliary position to the second auxiliary position can be directly obtained through the physical coordinates of the end effector at the first auxiliary position and the second auxiliary position, or obtained by converting the pixel coordinates of the calibration object into the physical coordinates of the end effector by using the translation calibration matrix; the translation transformation equation can represent the corresponding relationship of the above two ways of obtaining the physical coordinates, and therefore, according to the translation transformation equation, the pixel coordinates of the calibration object when the end effector is at the first auxiliary position, that is, the virtual coordinates of the calibration object located outside the field of view, can be obtained under the condition that the pixel coordinates of the calibration object in each auxiliary image, the physical coordinates of the end effector at the first auxiliary position and the second auxiliary position, and the intermediate matrix are known.

[0149] wherein the translation transformation equation is M*P'-M*P=W1-W2 when the camera is static, and the translation transformation equation is M t *P'-M t *P=W2-W1 when the camera is moving; wherein M is a translation calibration matrix as the intermediate matrix, P is the pixel coordinates of the calibration object in the auxiliary image, P' is the virtual coordinates, W1 is the physical coordinates of the end effector at the first auxiliary position, W2 is the physical coordinates of the end effector at the second auxiliary position, and M t is a generalized calibration matrix as the intermediate matrix.

[0150] It should be noted that the point or coordinate system transformation in two planes can be represented by a homogeneous linear transformation, that is, the translation, rotation, scaling and other transformations of the point or coordinate system can be represented by a set of homogeneous linear transformations T. When a pair of points or coordinate systems belonging to two planes satisfy the same linear transformation T, the physical coordinates of the to-be-solved point can be obtained by using the homogeneous transformation matrix acting on the pixel coordinates of the to-be-solved point; wherein the homogeneous transformation matrix can represent the transformation relationship of the homogeneous linear transformation T.

[0151] Specifically, in the embodiment, in the case that the camera is static, the physical coordinates of the end effector at the first auxiliary position W1 can be obtained by applying the translation calibration matrix M to the virtual coordinates P', and the physical coordinates of the end effector at the second auxiliary position W2 can be obtained by applying the translation calibration matrix M to the pixel coordinates P of the calibration object in the auxiliary image; in the case that the camera is moving, the physical coordinates of the end effector at the first auxiliary position W1 can be obtained by applying the generalized calibration matrix M to the virtual coordinates P', and the physical coordinates of the end effector at the second auxiliary position W2 can be obtained by applying the generalized calibration matrix M to the pixel coordinates P of the calibration object in the auxiliary image. t Specifically, in the embodiment, in the case that the camera is static, the physical coordinates of the end effector at the first auxiliary position W1 can be obtained by applying the translation calibration matrix M to the virtual coordinates P', and the physical coordinates of the end effector at the second auxiliary position W2 can be obtained by applying the translation calibration matrix M to the pixel coordinates P of the calibration object in the auxiliary image; in the case that the camera is moving, the physical coordinates of the end effector at the first auxiliary position W1 can be obtained by applying the generalized calibration matrix M to the virtual coordinates P', and the physical coordinates of the end effector at the second auxiliary position W2 can be obtained by applying the generalized calibration matrix M to the pixel coordinates P of the calibration object in the auxiliary image. t Specifically, in the embodiment, in the case that the camera is static, the physical coordinates of the end effector at the first auxiliary position W1 can be obtained by applying the translation calibration matrix M to the virtual coordinates P', and the physical coordinates of the end effector at the second auxiliary position W2 can be obtained by applying the translation calibration matrix M to the pixel coordinates P of the calibration object in the auxiliary image; in the case that the camera is moving, the physical coordinates of the end effector at the first auxiliary position W1 can be obtained by applying the generalized calibration matrix M to the virtual coordinates P', and the physical coordinates of the end effector at the second auxiliary position W2 can be obtained by applying the generalized calibration matrix M to the pixel coordinates P of the calibration object in the auxiliary image.

[0152] For the translation transformation equation M*P'-M*P=W1-W2, M*P' can represent the process of converting the pixel coordinates of the calibration object into the physical coordinates of the end effector by using the translation calibration matrix, at this time, the end effector is at the first auxiliary position; M*P can also represent the process of converting the pixel coordinates of the calibration object into the physical coordinates of the end effector by using the translation calibration matrix, at this time, the end effector is at the second auxiliary position; M*P'-M*P can represent the displacement of the end effector from the first auxiliary position to the second auxiliary position, and the displacement of the end effector from the first auxiliary position to the second auxiliary position can also be represented by the difference between the physical coordinates of the end effector at the first auxiliary position and the physical coordinates of the end effector at the second auxiliary position, that is, W1-W2.

[0153] It should be noted that, in the case that the camera is static, based on the projective transformation property, the translation calibration matrix M wherein s is the scaling factor between the camera coordinate system and the physical coordinate system, R represents the rotation matrix between the camera coordinate system and the physical coordinate system, t represents the translation matrix between the camera coordinate system and the physical coordinate system, and v represents the perspective transformation parameter between the camera coordinate system and the physical coordinate system; therefore, the translation transformation equation M*P'-M*P=W1-W2 can be converted into: wherein P x ' and P w ' are two-dimensional coordinate values of the virtual coordinates, P x and P y are two-dimensional coordinates of the pixel coordinates of the calibration object in the auxiliary image, W 1x and W 1y are two-dimensional coordinates of the physical coordinates of the end effector at the first auxiliary position, and W 2x and W 2y are two-dimensional coordinates of the physical coordinates of the end effector at the second auxiliary position. Based on the above formula, in the case that P x , P y , W 1x , W 1y, W 2x , W 2y , s, R, the two-dimensional coordinates of the virtual coordinates P x ' and P y ' can be obtained.

[0154] For the translation transformation equation M t *P'-M t *P=W2-W1, M t *P' can represent the process of converting the pixel coordinates of the calibration object into the physical coordinates of the end effector using the generalized calibration matrix, at this time, the end effector is located at the first auxiliary position; M t *P can also represent the process of converting the pixel coordinates of the calibration object into the physical coordinates of the end effector using the generalized calibration matrix, at this time, the end effector is located at the second auxiliary position; M t *P'-M t *P can represent the displacement of the end effector from the first auxiliary position to the second auxiliary position, and for the case of camera motion, the calibration object is in a static state, and the camera moves following the motion mechanism, so when the motion reference object is the camera, it can be considered that the movement of the calibration object relative to the camera is a reverse direction movement, therefore, the displacement of the end effector from the first auxiliary position to the second auxiliary position can also be represented by the difference between the physical coordinates of the end effector located at the second auxiliary position and the physical coordinates of the end effector located at the first auxiliary position, that is, W2-W1.

[0155] In the case of camera motion, based on the projective transformation property, the translation calibration matrix The generalized calibration matrix where s is the scaling factor between the camera coordinate system and the physical coordinate system, R represents the rotation matrix between the camera coordinate system and the physical coordinate system, t represents the translation matrix between the camera coordinate system and the physical coordinate system, v represents the perspective transformation parameter between the camera coordinate system and the physical coordinate system, and θ is the rotation angle of the rotational movement of the end effector, and R(θ) is the rotation component; therefore, the translation transformation equation M t *P'-M t *P=W2-W1 can be converted to: where P x ' and P y ' are the two-dimensional coordinate values of the virtual coordinates, P x and P y are the two-dimensional coordinates of the pixel coordinates of the calibration object in the auxiliary image, W 1x and W 1y are the two-dimensional coordinates of the physical coordinates of the end effector located at the first auxiliary position, W 2x and W 2yis a two-dimensional coordinate of the physical coordinate of the end effector located at the second auxiliary position. Based on the above formula, the two-dimensional coordinates P x , P y , W 1x , W 1y , W 2x , W 2y , s, R, R(θ) are known, the two-dimensional coordinates P x ' and P y ' of the virtual coordinates can be obtained.

[0156] In another implementation, the manner of modifying the translation calibration matrix can include the following steps:

[0157] The translation calibration matrix is modified by using the rotation angle of the rotational movement of the end effector and a projective transformation equation to obtain a generalized calibration matrix; wherein the projective transformation equation is an equation obtained based on the principle of projective transformation and is used to represent the transformation relationship between the rotation angle of the rotational movement of the end effector, the translation calibration matrix and the generalized calibration matrix.

[0158] It can be understood that the rotation angle of the rotational movement of the end effector can be taken as a parameter, represented in the matrix elements, and brought into the projective transformation equation obtained based on the principle of projective transformation to obtain the generalized calibration matrix.

[0159] The projective transformation equation is as follows: wherein θ represents the rotation angle of the rotational movement of the end effector, T x and T y are the displacement amounts of the parallel movement of the end effector from the first auxiliary position to the second auxiliary position, M t is the generalized calibration matrix as the intermediate matrix, and M is the translation calibration matrix as the intermediate matrix.

[0160] It can be understood that in the projective transformation equation, θ, M, T x and T y are known quantities, and therefore M t , that is, the generalized calibration matrix, can be obtained to complete the modification of the translation calibration matrix.

[0161] It should be noted that in the case where the rotation angle of the rotational movement of the end effector and the mechanism rotation center are the same each time, each displacement amount of the parallel movement of the end effector from each first auxiliary position to each second auxiliary position can be the same; in the case where the rotation angle of the rotational movement of the end effector and the mechanism rotation center are different each time, each displacement amount of the parallel movement of the end effector from each first auxiliary position to each second auxiliary position can be different.

[0162] In the embodiment, in the case of camera static and camera motion, the plurality of virtual coordinates can be determined by using the translation calibration matrix and the generalized calibration matrix respectively, and thus the method can be applied to various camera calibration scenes and has high portability.

[0163] In order to better understand the camera calibration method provided by the embodiment of the application, another embodiment is introduced as follows.

[0164] Figure 4 A flowchart of a camera calibration method provided by the embodiment of the application is shown in FIG. 4, which can include steps S401-S407. Figure 4

[0165] S401, controlling an end effector of a motion mechanism to perform multiple specified parallel movements so that a calibration object performs multiple parallel movements in a field of view of a camera, and controlling the camera to take pictures when the end effector moves to each target position, to obtain target images containing the calibration object.

[0166] S402, generating a translation calibration matrix based on pixel coordinates of the calibration object in each target image and physical coordinates of the end effector at each target position in a physical coordinate system.

[0167] S403, controlling the end effector to perform multiple rotational movements so that the end effector reaches multiple first auxiliary positions; wherein the first auxiliary positions are positions in which the calibration object is located outside the field of view of the camera.

[0168] S404, when reaching each first auxiliary position, controlling the end effector to perform parallel movement to move the end effector to a second auxiliary position, and controlling the camera to take pictures when the end effector moves to each second auxiliary position, to obtain auxiliary images containing the calibration object; wherein the second auxiliary positions are positions in which the calibration object is located within the field of view of the camera.

[0169] S405, determining a plurality of virtual coordinates by using pixel coordinates of the calibration object in each auxiliary image, physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and the translation calibration matrix; wherein the virtual coordinates are pixel coordinates of the calibration object in a camera coordinate system when the end effector is located at the first auxiliary position.

[0170] S406, performing fitting processing on the plurality of virtual coordinates and the physical coordinates of the end effector at each first auxiliary position with respect to a rotation center, to obtain a mechanism rotation center. ​

[0171] S407, determining the calibration result corresponding to the camera by using the mechanism rotation center and the translation calibration matrix.

[0172] It can be understood that the steps of the embodiment have been described in the foregoing embodiments, and will not be described in detail here.

[0173] Based on the above scheme, after the end effector moves by rotation, it reaches a plurality of first auxiliary positions such that the calibration object is located outside the field of view of the camera, and the pixel coordinates of the calibration object in the camera coordinate system, i.e., the virtual coordinates, are determined when the end effector is located at each first auxiliary position. Then, the plurality of virtual coordinates and the physical coordinates of the end effector located at each first auxiliary position are subjected to fitting processing with respect to the rotation center to obtain the mechanism rotation center. In this way, in the process of fitting the mechanism rotation center, the pixel coordinates of the calibration object can include the pixel coordinates outside the field of view of the camera, and the physical coordinates of the end effector can include the physical coordinates of the first auxiliary positions, which is equivalent to expanding the field of view of the camera. Therefore, by performing fitting processing on the pixel coordinates and the physical coordinates through the scheme, the sensitivity to the error of the physical coordinates or the pixel coordinates can be reduced, the stability of the obtained mechanism rotation center is improved, the error of the calibration result is reduced, and the accuracy of the camera calibration is improved.

[0174] In order to better understand how the method for camera calibration is implemented when the camera is static, another embodiment will be introduced as follows. As shown in FIG. 5, the method can include steps S501-S509: Figure 5

[0175] S501, controlling the end effector of the motion mechanism to move by a plurality of specified parallel movements to make the calibration object move by a plurality of parallel movements in the field of view of the camera, and controlling the camera to take a photograph when the end effector moves to each target position to obtain a target image containing the calibration object.

[0176] S502, based on the plurality of parallel movements and the target image, obtaining a feature point pixel coordinate set P and a motion mechanism physical coordinate set W; wherein the feature point pixel coordinate set P is a set including a plurality of pixel coordinates of the calibration object, and the motion mechanism physical coordinate set W is a set including a plurality of physical coordinates of the end effector.

[0177] It can be understood that the pixel coordinates of the calibration object can be the pixel coordinates of the feature points of the calibration object in the target image; and the physical coordinates of the end effector can be obtained by the control device in the process of controlling the motion mechanism to move.

[0178] ​S503, generating a translation calibration matrix according to an equation M*P=W, wherein M is the translation calibration matrix, P is the set of pixel coordinates of the feature points, and W is the set of physical coordinates of the motion mechanism.

[0179] S504, controlling the end effector to perform a plurality of rotational movements so that the end effector reaches a plurality of first auxiliary positions; wherein the first auxiliary position is a position in which the calibration object is located outside the field of view of the camera.

[0180] S505, at each time when reaching a first auxiliary position, controlling the end effector to perform a parallel movement so as to move the end effector to a second auxiliary position, and controlling the camera to take a picture when the end effector moves to each second auxiliary position, to obtain an auxiliary image containing the calibration object; wherein the second auxiliary position is a position in which the calibration object is located within the field of view of the camera.

[0181] S506, obtaining the physical coordinates W1 of the end effector when located at the first auxiliary position, the physical coordinates W2 of the end effector when located at the second auxiliary position, and the pixel coordinates P of the calibration object at the second auxiliary position.

[0182] S507, determining a plurality of virtual coordinates P' by using the physical coordinates W1, the physical coordinates W2, the pixel coordinates P, the translation calibration matrix M, and a translation transformation equation M*P'-M*P=W1-W2, under the condition that the camera is stationary.

[0183] S508, performing a fitting process on the plurality of virtual coordinates P' and the physical coordinates W1 of the end effector at each first auxiliary position with respect to a rotation center, to obtain a mechanism rotation center C.

[0184] S509, obtaining a target matrix M' by using the mechanism rotation center C, the translation calibration matrix M, and an equation M*C=0.

[0185] It can be understood that steps S501-S503 of the embodiment are similar to S301 of the foregoing embodiment, step S504 is the same as S302 of the foregoing embodiment, step S505 is the same as S303 of the foregoing embodiment, steps S506 and S507 have been described in the foregoing embodiment, and steps S508 and S509 are similar to S305 and S306 of the foregoing embodiment, and thus, no more detailed description is given herein.

[0186] Based on the above scheme, after the end effector moves by rotation, it reaches a plurality of first auxiliary positions such that the calibration object is located outside the field of view of the camera, and the pixel coordinates of the calibration object in the camera coordinate system, i.e., virtual coordinates, are determined when the end effector is located at each first auxiliary position. Then, fitting processing is performed on the plurality of virtual coordinates and the physical coordinates of the end effector at the respective first auxiliary positions with respect to the rotation center to obtain the mechanism rotation center. In this way, in the process of fitting the mechanism rotation center, the pixel coordinates of the calibration object can include the pixel coordinates outside the field of view of the camera, and the physical coordinates of the end effector can include the physical coordinates of the first auxiliary positions, which is equivalent to expanding the field of view of the camera. Therefore, by performing fitting processing on the pixel coordinates and the physical coordinates according to the scheme, the sensitivity to errors of the physical coordinates or the pixel coordinates can be reduced, the stability of the obtained mechanism rotation center is improved, the error of the calibration result is reduced, and the accuracy of camera calibration is improved.

[0187] To better understand how the method for camera calibration is implemented in the case of camera motion according to the embodiments of the present application, another embodiment will be introduced as follows, as shown in the method can include steps S601-S6011: Figure 6

[0188] S601, control the end effector of the motion mechanism to perform a plurality of specified parallel movements to make the calibration object move in the field of view of the camera a plurality of times, and control the camera to take a picture when the end effector moves to each target position, to obtain a target image containing the calibration object.

[0189] S602, based on the plurality of parallel movements and the target image, obtain a feature point pixel coordinate set P and a motion mechanism physical coordinate set W; wherein the feature point pixel coordinate set P is a set including a plurality of pixel coordinates of the calibration object, and the motion mechanism physical coordinate set W is a set including a plurality of physical coordinates of the end effector.

[0190] S603, generate a translation calibration matrix according to the equation M*P=W; wherein M is the translation calibration matrix, P is the feature point pixel coordinate set, and W is the motion mechanism physical coordinate set.

[0191] S604, control the end effector to perform a plurality of rotational movements to make the end effector reach a plurality of first auxiliary positions; wherein the first auxiliary positions are positions such that the calibration object is located outside the field of view of the camera.

[0192] ​S605, at each time the first auxiliary position is reached, controlling the end effector to move in parallel to move the end effector to a second auxiliary position, and controlling the camera to take a picture when the end effector moves to each second auxiliary position, to obtain an auxiliary image containing the calibration object; wherein the second auxiliary position is a position such that the calibration object is located in the field of view of the camera.

[0193] S606, obtaining the physical coordinates W1 of the end effector when located at the first auxiliary position, the physical coordinates W2 when located at the second auxiliary position, and the pixel coordinates P of the calibration object at the second auxiliary position.

[0194] S607, constructing a rotation matrix R(θ) based on the rotation angle θ of the rotation movement of the end effector; wherein the rotation matrix R(θ) is used to represent the transformation relationship between the camera coordinate system before rotation and the camera coordinate system after rotation.

[0195] It can be understood that the rotation matrix R(θ) can be an element containing θ in the aforementioned embodiment projection transformation equation, that is, Correspondingly, the form of the aforementioned embodiment projection transformation equation can be M t =M; wherein R(θ) can also be referred to as a rotation component, t is a translation matrix between the camera coordinate system and the physical coordinate system, and t can include T x and T y .

[0196] S608, using the rotation matrix R(θ) to obtain a generalized calibration matrix M t for camera motion.

[0197] It can be understood that based on the equation , the following can be obtained Therefore, the generalized calibration matrix M t for camera motion can be obtained.

[0198] S609, in the case of camera motion, using the physical coordinates W1, the physical coordinates W2, the pixel coordinates P, the generalized calibration matrix M t , and the translation transformation equation M t *P'-M t *P=W1-W2, to determine a plurality of virtual coordinates P'.

[0199] S6010, performing fitting processing on the plurality of virtual coordinates P' and the physical coordinates W1 of the end effector located at each first auxiliary position about the rotation center, to obtain the mechanism rotation center C.

[0200] S6011, obtaining a target matrix M' by using the mechanism rotation center C, the translation calibration matrix M, and the equation M*C=0.

[0201] It can be understood that steps S601-S603 of the embodiment are similar to S301 in the foregoing embodiment, step S604 is the same as S302 in the foregoing embodiment, step S605 is the same as S303 in the foregoing embodiment, steps S606, S607, S608, and S609 have been described in the foregoing embodiment, and steps S6010 and S6011 are similar to S305 and S306 in the foregoing embodiment, and thus, no more detailed description is given herein.

[0202] Based on the above scheme, after the end effector moves by rotation, the end effector reaches a plurality of first auxiliary positions at which the calibration object is located outside the field of view of the camera, and the pixel coordinates of the calibration object in the camera coordinate system are determined when the end effector is located at each first auxiliary position, that is, virtual coordinates. Then, fitting processing is performed on the plurality of virtual coordinates and the physical coordinates of the end effector at the respective first auxiliary positions with respect to the rotation center to obtain the mechanism rotation center. In this way, in the process of fitting the mechanism rotation center, the pixel coordinates of the calibration object can include the pixel coordinates outside the field of view of the camera, and the physical coordinates of the end effector can include the physical coordinates of the first auxiliary positions, which is equivalent to expanding the field of view of the camera. Therefore, by performing fitting processing on the pixel coordinates and the physical coordinates, the sensitivity to errors of the physical coordinates or the pixel coordinates can be reduced, the stability of the obtained mechanism rotation center is improved, the error of the calibration result is reduced, and the accuracy of camera calibration is improved.

[0203] Figure 7 A structural schematic diagram of a device for camera calibration provided by an embodiment of the present application is shown in FIG. 7, which can include the following modules. Figure 7

[0204] The construction module 710 is configured to construct a translation calibration matrix corresponding to a camera based on a plurality of specified parallel movements of an end effector of a motion mechanism, wherein the plurality of specified parallel movements cause a calibration object to move in parallel multiple times within a field of view of the camera.

[0205] The first control module 720 is configured to control the end effector to move by rotation multiple times to cause the end effector to reach a plurality of first auxiliary positions, wherein the first auxiliary positions are positions at which the calibration object is located outside the field of view of the camera.

[0206] ​The second control module 730 is configured to control the end effector to move in parallel to move the end effector to a second auxiliary position each time the first auxiliary position is reached, and control the camera to take a picture when the end effector moves to each second auxiliary position, to obtain an auxiliary image containing the calibration object; wherein the second auxiliary position is a position such that the calibration object is located in the field of view of the camera.

[0207] The first determination module 740 is configured to determine a plurality of virtual coordinates by using the pixel coordinates of the calibration object in each auxiliary image, the physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and the translation calibration matrix; wherein the virtual coordinates are pixel coordinates of the calibration object in the camera coordinate system when the end effector is located at the first auxiliary position.

[0208] The fitting module 750 is configured to perform fitting processing on the plurality of virtual coordinates and the physical coordinates of the end effector at each first auxiliary position with respect to a rotation center, to obtain a mechanism rotation center.

[0209] The second determination module 760 is configured to determine a calibration result corresponding to the camera by using the mechanism rotation center and the translation calibration matrix.

[0210] The first determination module 740 can include:

[0211] The first determination unit is configured to determine a plurality of virtual coordinates by using the pixel coordinates of the calibration object in each auxiliary image, the physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and an intermediate matrix; wherein, in the case that the camera is stationary, the intermediate matrix is the translation calibration matrix, and in the case that the camera moves, the intermediate matrix is a generalized calibration matrix obtained by performing correction processing on the translation calibration matrix, and the correction processing is used to convert the translation calibration matrix into a generalized calibration matrix representing the transformation relationship between a rotated camera coordinate system and a physical coordinate system, wherein the rotated camera coordinate system is a camera coordinate system formed after the camera follows the moving mechanism to move by rotation.

[0212] The mechanism rotation center includes a pixel rotation center in the camera coordinate system and a physical rotation center in the physical coordinate system.

[0213] The second determination module 760 can include:

[0214] The second determining unit is configured to normalize the translation calibration matrix by using the pixel rotation center, to obtain a calibration result corresponding to the camera; wherein the normalization is configured to convert the translation calibration matrix into a matrix when an origin of a physical coordinate system corresponding to the motion mechanism is translated to the physical rotation center.

[0215] The first determining unit is configured to:

[0216] According to a translation transformation equation, a plurality of virtual coordinates are determined by using pixel coordinates of the calibration object in each auxiliary image, physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and an intermediate matrix; wherein the translation transformation equation is an equation obtained based on a principle of plane Euclidean transformation, and is used to represent a transformation relationship between the pixel coordinates of the calibration object in the auxiliary image, the physical coordinates of the end effector at the first auxiliary position and second auxiliary position, the intermediate matrix, and the virtual coordinates.

[0217] Wherein, in the case that the camera is static, the translation transformation equation is: M*P'-M*P=W1-W2, and in the case that the camera is moving, the translation transformation equation is: M t *P'-M t *P=W2-W1; wherein M is a translation calibration matrix as the intermediate matrix, P is the pixel coordinates of the calibration object in the auxiliary image, P' is the virtual coordinates, W1 is the physical coordinates of the end effector at the first auxiliary position, W2 is the physical coordinates of the end effector at the second auxiliary position, and M t is a generalized calibration matrix as the intermediate matrix.

[0218] The manner of correcting the translation calibration matrix includes:

[0219] A generalized calibration matrix is obtained by correcting the translation calibration matrix by using a rotation angle of the end effector in the rotation movement and a projective transformation equation.

[0220] Wherein, the projective transformation equation is an equation obtained based on a principle of projective transformation, and is used to represent a transformation relationship between the rotation angle of the end effector in the rotation movement, the translation calibration matrix, and the generalized calibration matrix.

[0221] The projective transformation equation is: Wherein, θ represents the rotation angle of the end effector in the rotation movement, T x and T y are displacement amounts of the end effector in parallel movement from the first auxiliary position to the second auxiliary position, M tM is a translation calibration matrix as the intermediate matrix.

[0222] The second determining unit can include:

[0223] The second determining sub-unit is configured to use the equation M*C=0 to represent the translation component in the translation calibration matrix by a rotation angle of the rotation center of the pixels and the rotation movement of the end effector, wherein C is a coordinate of the rotation center of the pixels, and M is the translation calibration matrix.

[0224] The construction module 710 can include:

[0225] The parallel movement unit is configured to control the end effector of the motion mechanism to perform multiple specified parallel movements to make the calibration object perform multiple parallel movements in the field of view of the camera.

[0226] The photographing unit is configured to control the camera to take a photograph when the end effector moves to each target position, to obtain a target image containing the calibration object.

[0227] The generation unit is configured to generate a translation calibration matrix based on pixel coordinates of the calibration object in each target image and physical coordinates of the end effector in each target position in a physical coordinate system.

[0228] According to the above scheme, after the end effector moves by rotation, the end effector reaches multiple first auxiliary positions at which the calibration object is located outside the field of view of the camera, and the pixel coordinates of the calibration object in the camera coordinate system, i.e., virtual coordinates, are determined when the end effector is located at each first auxiliary position. Then, the multiple virtual coordinates and the physical coordinates of the end effector at each first auxiliary position are fitted with respect to the rotation center to obtain the mechanism rotation center. In this way, in the process of fitting the mechanism rotation center, the pixel coordinates of the calibration object can include the pixel coordinates outside the field of view of the camera, and the physical coordinates of the end effector can include the physical coordinates of the first auxiliary positions, which is equivalent to expanding the field of view of the camera. Therefore, by fitting the pixel coordinates and the physical coordinates according to the scheme, the sensitivity to errors of the physical coordinates or the pixel coordinates can be reduced, the stability of the obtained mechanism rotation center is improved, the error of the calibration result is reduced, and the accuracy of the camera calibration is improved.

[0229] The embodiments of the present application also provide an electronic device, as shown in Figure 8 The electronic device includes

[0230] The memory 801 is configured to store a computer program.

[0231] The processor 802 is configured to execute the program stored in the memory 801 to implement the following steps:

[0232] a translation matrix corresponding to the camera is constructed based on a plurality of specified parallel movements of the end effector of the motion mechanism, wherein the plurality of specified parallel movements cause the calibration object to move in parallel a plurality of times within the field of view of the camera;

[0233] the end effector is controlled to move in rotation a plurality of times so that the end effector reaches a plurality of first auxiliary positions, wherein the first auxiliary positions are positions such that the calibration object is located outside the field of view of the camera;

[0234] at each time when a first auxiliary position is reached, the end effector is controlled to move in parallel so as to move the end effector to a second auxiliary position, and the camera is controlled to take a picture when the end effector moves to each second auxiliary position, to obtain an auxiliary image containing the calibration object, wherein the second auxiliary position is a position such that the calibration object is located within the field of view of the camera;

[0235] a plurality of virtual coordinates are determined by using pixel coordinates of the calibration object in each auxiliary image, physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and the translation matrix, wherein the virtual coordinates are pixel coordinates of the calibration object in the camera coordinate system when the end effector is located at the first auxiliary position;

[0236] a fitting process is performed on the plurality of virtual coordinates and the physical coordinates of the end effector at each first auxiliary position with respect to a rotation center, to obtain a mechanism rotation center;

[0237] a calibration result corresponding to the camera is determined by using the mechanism rotation center and the translation matrix.

[0238] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0239] The communication interface is used for communication between the electronic device and other devices.

[0240] The memory can include a random access memory (RAM) and can also include a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0241] The processor described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0242] In yet another embodiment provided in the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of any of the camera calibration methods described above.

[0243] In yet another embodiment provided in the present application, a computer program product containing instructions, which, when run on a computer, causes the computer to execute the camera calibration method of any of the above embodiments.

[0244] In the embodiments described above, all or some of the steps can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, all or some of the steps can be implemented in the form of one or more computer programs or program elements. The computer programs reside (at least temporarily) in a memory of a computer during execution. The memory can be a RAM memory, a flash memory, a ROM memory, an EPROM memory, or any other suitable memory. The memory can be integral to or separate from the computer. The computer programs can be written in any suitable programming language, such as C, C++, Java, Visual Basic, etc. The computer programs can be written in assembly or machine language, if desired. The computer programs can be distributed over network coupled file servers, or can be distributed by any other suitable means.

[0245] It is to be understood that the terminology "first", "second", etc. used herein merely for the purpose of distinguishing one entity or action from another, and does not necessarily imply these entities or actions are mutually exclusive or are in a certain sequence. Furthermore, the terms "comprise", "comprising", "include", "including", or any other variant are intended to cover a non-exclusive inclusion, such that processes, methods, articles, or apparatuses that comprise, include, or are including entities listed thereafter are not limited to those entities but can also include other entities not expressly listed or inherent to such processes, methods, articles, or apparatuses.

[0246] Each of the embodiments described in the specification adopt a relevant manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0247] The above merely provides the preferred embodiment of the present application, and not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for camera calibration, characterized in that, The method is applied to a control device, and comprises: constructing a translation calibration matrix corresponding to a camera based on a plurality of specified parallel movements of an end effector of a motion mechanism, wherein the plurality of specified parallel movements cause a calibration object to move in parallel a plurality of times within a field of view of the camera; controlling the end effector to move in rotation a plurality of times so that the end effector reaches a plurality of first auxiliary positions, wherein the first auxiliary positions are positions at which the calibration object is located outside the field of view of the camera; controlling the end effector to move in parallel each time the first auxiliary position is reached, so as to move the end effector to a second auxiliary position, and controlling the camera to take a photograph each time the end effector moves to each second auxiliary position, to obtain an auxiliary image containing the calibration object, wherein the second auxiliary positions are positions at which the calibration object is located within the field of view of the camera; determining a plurality of virtual coordinates by using pixel coordinates of the calibration object in each auxiliary image, physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and the translation calibration matrix, wherein the virtual coordinates are pixel coordinates of the calibration object in a camera coordinate system when the end effector is located at the first auxiliary position; performing fitting processing on the plurality of virtual coordinates and the physical coordinates of the end effector at each first auxiliary position with respect to a rotation center, to obtain a mechanism rotation center; determining a calibration result corresponding to the camera by using the mechanism rotation center and the translation calibration matrix.

2. The method of claim 1, wherein, The method for determining a plurality of virtual coordinates by using pixel coordinates of the calibration object in each auxiliary image, physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and the translation calibration matrix comprises: The method for determining a plurality of virtual coordinates by using pixel coordinates of the calibration object in each auxiliary image, physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and an intermediate matrix comprises: determining a plurality of virtual coordinates by using pixel coordinates of the calibration object in each auxiliary image, physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and an intermediate matrix, wherein, in the case that the camera is stationary, the intermediate matrix is the translation calibration matrix, and in the case that the camera moves, the intermediate matrix is a generalized calibration matrix obtained by performing correction processing on the translation calibration matrix, and the correction processing is used to convert the translation calibration matrix into a generalized calibration matrix representing a transformation relationship between a rotated camera coordinate system and a physical coordinate system, wherein the rotated camera coordinate system is a camera coordinate system formed after the camera follows the motion mechanism to move in rotation.

3. The method of claim 1, wherein, The mechanism rotation center comprises a pixel rotation center in a camera coordinate system and a physical rotation center in a physical coordinate system. The method for determining a calibration result corresponding to the camera by using the mechanism rotation center and the translation calibration matrix comprises: The method for determining a calibration result corresponding to the camera by using the mechanism rotation center and the translation calibration matrix comprises: performing normalization processing on the translation calibration matrix by using the pixel rotation center, to obtain the calibration result corresponding to the camera, wherein the normalization processing is used to convert the translation calibration matrix into a matrix in the case that an origin of a physical coordinate system corresponding to the motion mechanism is translated to the physical rotation center.

4. The method of claim 2, wherein, The method comprises the following steps: According to a translation transformation equation, a plurality of virtual coordinates are determined by using the pixel coordinates of the calibration object in each auxiliary image, the physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and an intermediate matrix; wherein the translation transformation equation is an equation obtained based on a plane Euclidean transformation principle and is used to represent a transformation relationship between the pixel coordinates of the calibration object in the auxiliary image, the physical coordinates of the end effector at the first auxiliary position and second auxiliary position, the intermediate matrix, and the virtual coordinates.

5. The method of claim 4, wherein, The translation transformation equation is M * P' - M * P = W1 - W2 in the case of a stationary camera, and M t * P' - M t * P = W2 - W1 in the case of a moving camera. wherein M is a translation calibration matrix as the intermediate matrix, P is a pixel coordinate of the calibration object in the auxiliary image, P' is the virtual coordinate, W1 is a physical coordinate of the end effector at the first auxiliary position, W2 is a physical coordinate of the end effector at the second auxiliary position, M t is a generalized calibration matrix as the intermediate matrix.

6. The method of claim 2, wherein, The method for correcting the translation calibration matrix comprises the following steps: The translation calibration matrix is corrected by using a rotation angle of the end effector during the rotation movement and a projective transformation equation to obtain a generalized calibration matrix; wherein the projective transformation equation is an equation obtained based on a projective transformation principle and is used to represent a transformation relationship between the rotation angle of the end effector during the rotation movement, the translation calibration matrix, and the generalized calibration matrix.

7. The method of claim 6, wherein, The projective transformation equation is: wherein θ represents a rotation angle of the end effector to perform a rotation movement, T x and T y is a displacement amount of the end effector to move in parallel from the first auxiliary position to the second auxiliary position, M t is a generalized calibration matrix as the intermediate matrix, and M is a translation calibration matrix as the intermediate matrix.

8. The method of claim 3, wherein, The method for normalizing the translation calibration matrix by using the pixel rotation center comprises the following steps: The translation component in the translation calibration matrix is represented by the pixel rotation center and the rotation angle of the end effector during the rotation movement by using an equation M*C=0; wherein C is the coordinates of the pixel rotation center, and M is the translation calibration matrix.

9. The method according to any one of claims 1 to 8, characterized in that, The method for constructing a translation calibration matrix corresponding to a camera based on a plurality of specified parallel movements of an end effector of a motion mechanism comprises the following steps: The end effector of the motion mechanism is controlled to perform a plurality of specified parallel movements so that the calibration object performs a plurality of parallel movements in the field of view of the camera; The camera is controlled to take pictures when the end effector moves to each target position to obtain target images containing the calibration object; A translation calibration matrix is generated based on the pixel coordinates of the calibration object in each target image and the physical coordinates of the end effector at each target position in a physical coordinate system.

10. An apparatus for camera calibration, the apparatus comprising: The device comprises: A construction module is configured to construct a translation calibration matrix corresponding to a camera based on a plurality of specified parallel movements of an end effector of a motion mechanism; wherein the plurality of specified parallel movements cause the calibration object to perform a plurality of parallel movements in the field of view of the camera; A first control module is configured to control the end effector to perform a plurality of rotation movements so that the end effector reaches a plurality of first auxiliary positions; wherein the first auxiliary positions are positions at which the calibration object is located outside the field of view of the camera; A second control module is configured to control the camera to take pictures when the end effector moves to each target position to obtain target images containing the calibration object; The second control module is configured to control the end effector to move in parallel to move the end effector to a second auxiliary position each time the first auxiliary position is reached, and control the camera to take a picture when the end effector moves to each second auxiliary position, to obtain an auxiliary image containing the calibration object; wherein the second auxiliary position is a position such that the calibration object is located in the field of view of the camera. The first determination module is configured to determine a plurality of virtual coordinates by using pixel coordinates of the calibration object in each auxiliary image, physical coordinates of the end effector at each first auxiliary position and second auxiliary position, and the translation calibration matrix; wherein the virtual coordinates are pixel coordinates of the calibration object in the camera coordinate system when the end effector is located at the first auxiliary position. The fitting module is configured to perform fitting processing on the plurality of virtual coordinates and the physical coordinates of the end effector at each first auxiliary position with respect to a rotation center, to obtain a mechanism rotation center. The second determination module is configured to determine a calibration result corresponding to the camera by using the mechanism rotation center and the translation calibration matrix.

11. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method in any one of claims 1-9. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method in any one of claims 1-9. ​ 12. A computer-readable storage medium, characterized in that, ​

Images