Calibration Method, System, Device and Storage Medium of Galvanometer Scanning Device

Abstract

The present invention provides a calibration method, system, device and storage medium for a galvanometer scanning device. The galvanometer scanning device includes a light source, a camera and a galvanometer, and comprises: obtaining a pre-established galvanometer scanning model, the galvanometer scanning model being established based on a reflected light plane formed by the galvanometer reflecting incident light, the reflected light plane being associated with the phase information and having an intersection point with the rotation axis of the galvanometer; optimizing and solving the galvanometer scanning model according to the phase values Φ corresponding to the three-dimensional coordinates (x C , y C , z C ) of multiple points to generate model parameters; determining the coordinates of the object point according to the function associated with the reflected light plane and the phase value Φ and the straight line formed by the connection line between the image point in the camera and the camera optical center point. In the present invention, the three-dimensional shape information of the object to be measured can be quickly calculated, and compared with the calibration method based on the complete coaxiality of the rotation axis and the galvanometer, the calculation error caused by the non-coaxiality of the rotation axis and the galvanometer during installation is avoided.

Description

Background Art

[0002] In the process of image measurement and machine vision applications, in order to determine the mutual relationship between the three-dimensional geometric position of a certain point on the surface of a spatial object and its corresponding point in the image, it is necessary to establish a geometric model of camera imaging, and these geometric model parameters are camera parameters. Under most conditions, these camera parameters must be obtained through experiments and calculations, and this process of solving parameters is called camera calibration. Whether in image measurement or machine vision applications, the calibration of camera parameters is a very crucial link, and the accuracy of its calibration results and the stability of the algorithm directly affect the accuracy of the results generated by the camera's work.

[0003] The laser structured light imaging principle based on a galvanometer scanning device is essentially no different from the traditional DLP structured light imaging principle, both are based on the triangulation ranging principle. However, the slight difference is that the DLP projector itself is an imaging system and can be directly analyzed using the pinhole imaging model for binocular stereo vision matching; while the galvanometer scanning device is not an imaging system, and only by relating the deflection angle of the galvanometer to the imaging position information in the encoded structured light can three-dimensional reconstruction be carried out. The calibration of the galvanometer scanning device is a key step to ensure measurement accuracy. In the prior art, it often relies on high-precision calibration devices, and the actual operation process of the calibration method is complex and the calibration speed is slow, making it difficult to meet the requirements of on-site rapid calibration in actual production. Summary of the Invention

[0004] Aiming at the deficiencies in the prior art, the purpose of the present invention is to provide a calibration method, system, device and storage medium for a galvanometer scanning device.

[0005] According to the calibration method of the galvanometer scanning device provided by the present invention, the galvanometer scanning device includes a light source, a camera and a galvanometer, and the method includes the following steps:

[0006] Obtain the correlation relationship between the phase information and the deflection angle, where the phase information is the phase of the light intensity of the incident light projected by the light source changing with time, and the deflection angle is the deflection angle of the galvanometer changing with time;

[0007] Obtain a pre-established galvanometer scanning model, which is established based on the reflected light plane formed by the galvanometer reflecting the incident light, and the reflected light plane is associated with the phase information and has an intersection with the rotation axis of the galvanometer;

[0008] Obtain the three-dimensional coordinates (x C , y C , z C ) of multiple points on the calibration plate image and the corresponding phase value Φ, according to the three-dimensional coordinates (x C , y C , z C)The corresponding phase value Φ optimizes and solves the galvanometer scanning model to generate model parameters;

[0009] Determine the function of the reflected light plane associated with the phase value Φ according to the model parameters, and determine the coordinates of the object point according to the function of the reflected light plane associated with the phase value Φ and the straight line formed by the connection line between the image point and the camera optical center point in the camera.

[0010] Preferably, when establishing the correlation relationship between the phase information and the deflection angle, the following steps are included:

[0011] Obtain the projection light intensity relationship I n (t), where the projection light intensity relationship I n (t) is the relationship between the light intensity of the incident light projected by the light source and time;

[0012] Obtain the multi-step phase shift relationship, where the multi-step phase shift relationship is the relationship between the phase Φ(t) and the light intensity of the incident light. Generate the phase-time relationship between the phase Φ(t) and time t according to the projection light intensity relationship and the multi-step phase shift relationship;

[0013] Obtain the galvanometer time relationship between the deflection angle of the galvanometer and time, and generate the correlation relationship between the deflection angle and the phase according to the galvanometer time relationship and the phase-time relationship.

[0014] Preferably, when establishing the galvanometer scanning model, the following steps are included:

[0015] Obtain the normal vector of the incident light plane pre-established in the camera coordinate system and the intersection coordinates (x0, y0, z0) between the incident light plane and the rotation axis of the galvanometer;

[0016] Obtain the temporary coordinate system pre-established on the initial plane of the galvanometer, generate the normal vector of the galvanometer plane based on the temporary coordinate system, and generate the expression of the normal vector of the galvanometer plane in the camera coordinate system according to the rotation matrix between the temporary coordinate system and the camera coordinate system

[0017] According to the normal vector of the incident light plane and the normal vector of the galvanometer plane generate the normal vector of the reflected light plane The normal vector of the galvanometer plane is associated with the deflection angle;

[0018] According to the normal vector of the reflected light plane Establish the equation of the reflected light plane based on the intersection point coordinates (x0, y0, z0), and establish the galvanometer scanning model according to the equation of the reflected light plane.

[0019] Preferably, when solving for the generation model parameters, the following steps are included:

[0020] Obtain a pre-established camera model, where the camera model includes the relationship between the pixel coordinate system, the camera coordinate system, and the world coordinate system;

[0021] Obtain multiple calibration plate images, and based on the camera model, determine the three-dimensional coordinates of the object points corresponding to the image points on the calibration plate images in the camera coordinate system;

[0022] Obtain the phase-shift encoded image corresponding to each calibration plate image, decode the phase-shift encoded image, generate the phase value Φ of each image point in the calibration plate image, and determine the corresponding relationship between the three-dimensional coordinates (x C , y C , z C ) of multiple object points on the calibration plate and the phase value Φ;

[0023] According to the phase values Φ corresponding to the three-dimensional coordinates (x C , y C , z C ) of multiple points, optimize and solve the galvanometer scanning model to generate the model parameters.

[0024] Preferably, when determining the coordinates of the object points, the following steps are included:

[0025] Determine the galvanometer scanning model according to the model parameters, that is, determine the function associated with the reflected light plane and the phase value Φ;

[0026] Obtain the target line, which is generated according to the connection line between the image point and the camera optical center point in the camera. Determine the light intensity of the image point according to the pixel value of the image point, and determine the phase value Φ corresponding to the image point according to the multi-step phase shift relationship;

[0027] Determine the reflected light plane according to the phase value Φ, so that the intersection point of the target line and the reflected light plane can be determined as the object point on the target, and the corresponding object point coordinates are generated.

[0028] Preferably, in the temporary coordinate system, the rotation axis of the galvanometer is used as the Z G axis, the unit vector perpendicular to the Z G axis on the initial galvanometer plane is used as the X G axis, the intersection point of the X G axis and the Z G axis on the initial galvanometer plane is used as the origin O G , and the perpendicular to the Z G axis and the XG The unit vector of the axis is Y G axis.

[0029] Preferably, generating the expression of the normal vector of the galvanometer plane in the camera coordinate system includes the following steps:

[0030] Obtain the rotation axis vector of the galvanometer in the camera coordinate system and the normal vector of the galvanometer plane in the temporary coordinate system;

[0031] Generate a rotation matrix from the temporary coordinate system to the camera coordinate system according to the rotation axis vector of the galvanometer and the normal vector of the galvanometer plane;

[0032] Convert the normal vector of the galvanometer plane in the temporary coordinate system to the normal vector of the galvanometer plane in the camera coordinate system according to the rotation matrix

[0033] According to the calibration system of the galvanometer scanning device provided by the present invention, the galvanometer scanning device includes a light source, a camera and a galvanometer, and includes the following modules:

[0034] An association information module, configured to obtain the association relationship between the phase information and the deflection angle, where the phase information is the phase of the light intensity of the incident light projected by the light source changing with time, and the deflection angle is the deflection angle of the galvanometer changing with time;

[0035] A model calling module, configured to obtain a pre-established galvanometer scanning model, where the galvanometer scanning model is established based on the reflection light plane formed by the galvanometer reflecting the reflected light, the reflection light plane is associated with the phase information and has an intersection with the rotation axis of the galvanometer;

[0036] A model optimization module, configured to obtain the three-dimensional coordinates (x C , y C , z C ) of multiple points on the calibration plate image and the corresponding phase value Φ, and optimize and solve the galvanometer scanning model according to the phase value Φ corresponding to the three-dimensional coordinates (x C , y C , z C ) to generate model parameters;

[0037] An object point determination module, configured to determine the function associated with the phase value Φ of the reflection light plane according to the model parameters, and determine the coordinates of the object point according to the function associated with the phase value Φ of the reflection light plane and the straight line formed by the connection line between the image point and the camera optical center point in the camera.

[0038] According to the calibration device of the galvanometer scanning device provided by the present invention, it includes:

[0039] A processor;

[0040] A memory module in which executable instructions of the processor are stored;

[0041] Wherein, the processor is configured to execute the steps of the calibration method of the galvanometer scanning device by executing the executable instructions.

[0042] A computer-readable storage medium provided according to the present invention is used to store a program, and when the program is executed, the steps of the calibration method of the galvanometer scanning device are implemented.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] In the present invention, a galvanometer scanning model is established based on the intersection of the reflected light plane and the rotation axis of the galvanometer, and the model parameters are solved to generate a function that associates the reflected light plane with the phase value Φ. Then, for an image point, a target line can be determined by connecting the point to the camera optical center point, and the reflected light plane corresponding to the phase value can be obtained. The three-dimensional coordinates of the object point on the target are determined according to the intersection of the target line and the reflected light plane, realizing the rapid calculation of the three-dimensional shape information of the object to be measured. Compared with the calibration method based on the complete coaxiality of the rotation axis and the galvanometer, the calculation error caused by the non-coaxial installation of the rotation axis and the galvanometer is avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, objects, and advantages of the present invention will become more obvious:

[0046] Figure 1 Schematic diagram of the working principle of the galvanometer scanning device in the embodiment of the present invention;

[0047] Figure 2 Flowchart of the steps of the calibration method of the galvanometer scanning device in the embodiment of the present invention;

[0048] Figure 3 Flowchart of the steps of the association between the deflection angle and the phase in the embodiment of the present invention;

[0049] Figure 4 Flowchart of the steps of establishing the galvanometer scanning model in the embodiment of the present invention;

[0050] Figure 5 Schematic diagram of the establishment of the pixel, camera, galvanometer, and world coordinate system in the embodiment of the present invention;

[0051] Figure 6 This is the flowchart of the steps for generating the normal vector of the galvanometer plane in the embodiments of the present invention;

[0052] Figure 7 This is the flowchart of the steps for generating model parameters in the embodiments of the present invention;

[0053] Figure 8 This is the flowchart of the steps for determining the coordinates of the object point in the embodiments of the present invention;

[0054] Figure 9 This is the schematic diagram of the working principle of the calibration of the galvanometer scanning model in the embodiments of the present invention;

[0055] Figure 10 This is the schematic diagram of the control logic of the galvanometer scanning model in the embodiments of the present invention;

[0056] Figure 11 This is the schematic diagram of the modules of the calibration system of the galvanometer scanning device in the embodiments of the present invention;

[0057] Figure 12 This is the schematic diagram of the structure of the calibration device of the galvanometer scanning device in the embodiments of the present invention;

[0058] Figure 13 This is the schematic diagram of the structure of the article picking system in the embodiments of the present invention, and

[0059] Figure 14 This is the schematic diagram of the structure of the computer-readable storage medium in the embodiments of the present invention.

[0060] In the figure:

[0061] 1 is a camera; 2 is a light source; 3 is a galvanometer; 4 is a calibration plate; 100 is a depth camera; 200 is a storage unit; 300 is a feeding unit; 400 is a robot unit. Detailed implementation manners

[0062] The present invention will be described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made. These all belong to the protection scope of the present invention.

[0063] It should be noted that when an element is referred to as "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, the connection can be for fixing or for circuit connection.

[0064] It should be understood that the orientation or positional relationship indicated by the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the embodiments of the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention.

[0065] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present invention, "a plurality" means two or more, unless otherwise specifically defined.

[0066] Figure 1 is a schematic diagram of the working principle of the galvanometer scanning device in the embodiment of the present invention, as Figure 1 shown, the galvanometer scanning device includes a light source 2, a camera 1, and a galvanometer 3; the light source 2 generally uses a line laser, and the line laser emits linear laser; the laser is reflected by the galvanometer 3 and projected onto the surface of the target object, and the laser deflects at an angle as the galvanometer 3 rotates, thereby completing the scanning of the object surface and forming a preset pattern in a grating shape on the object surface.

[0067] Figure 2 is a flowchart of the steps of the calibration method of the galvanometer scanning device in the embodiment of the present invention, as Figure 2 shown, the calibration method of the galvanometer scanning device provided by the present invention includes the following steps:

[0068] Step S1: Obtain the correlation relationship between the phase information and the deflection angle, where the phase information is the phase of the light intensity of the incident light projected by the light source changing with time, and the deflection angle is the deflection angle of the galvanometer changing with time;

[0069] Figure 3 is a flowchart of the steps of the correlation between the deflection angle and the phase in the embodiment of the present invention, as Figure 3As shown, in the embodiment of the present invention, a coding method for Gray code-assisted phase-shift unwrapping imaging is adopted to establish the relationship between phase information and deflection angle. The step S1 includes the following steps:

[0070] Step S101: Obtain the projected light intensity relationship I n (t), where the projected light intensity relationship I n (t) is the relationship between the light intensity of the incident light projected by the light source and time;

[0071] When projecting the nth phase-shift pattern, the light intensity of the incident light projected by the laser changes sinusoidally with time. The projected light intensity relationship I n (t) is:

[0072]

[0073] In the formula, I(t) is the projected light intensity at time t, i′(t) is the bias light intensity at time t, i″(t) is the modulation light intensity at time t, f L is the modulation frequency, Φ0 = 0 is the initial phase, n is the serial number of the phase-shift pattern, and N = 16 is the number of phase-shift patterns.

[0074] Step S102: Obtain the multi-step phase-shift relationship, which is the relationship between the phase Φ(t) and the light intensity of the incident light. Generate the phase-time relationship between the phase Φ(t) and time t according to the projected light intensity relationship and the multi-step phase-shift relationship;

[0075] According to the multi-step phase-shift relationship, the relationship between the phase Φ(t) and time t can be obtained:

[0076]

[0077] In the formula, Φ(t) is the phase at time t.

[0078] Step S103: Obtain the galvanometer time relationship between the deflection angle of the galvanometer and time. Generate the correlation relationship between the deflection angle and the phase according to the galvanometer time relationship and the phase-time relationship.

[0079] During the process of projecting a single pattern, the galvanometer rotates linearly with time, and the rotation angle is:

[0080] α(t) = k α t + α0

[0081] In the formula, α(t) is the rotation angle at time t, k α is the rotational angular velocity, and α0 is the initial angle;

[0082] Such as Figure 1As shown, the imaging of point P on the surface of the target object and the image point Q on the camera imaging plane C are a pair of imaging points. On the premise that the camera model is regarded as a pinhole imaging model, the line connecting the image point Q C and the camera optical center point O must pass through the object point P. Therefore, a straight line can be determined according to the image point Q C and the camera optical center point O. Moreover, according to the pixel value of the image point Q C , the light intensity irradiating on the image point Q C can be obtained. Given the projected light intensity I(t), the phase Φ(t) can be calculated. The phase Φ(t) and the angle α(t) are related by time t

[0083] α(t) = k Φ2α Φ(t) + α0

[0084] In the formula, if α(t) is determined, the reflection light plane corresponding to this angle can be obtained, and this reflection light plane is the plane illuminating the object point P. Determining a straight line and a plane passing through point P can uniquely determine the position of the object point P.

[0085] In the embodiment of the present invention, the phase and the rotation angle of the galvanometer are correlated through the time relationship to form a continuous linear correlation relationship, so that the phase value at any time can be obtained, improving the calculation accuracy of the rotation angle of the galvanometer, and further improving the accuracy of the entire calibration algorithm.

[0086] Step S2: Obtain a pre-established galvanometer scanning model, which is established based on the reflection light plane formed by the reflection of the galvanometer. The reflection light plane is associated with the phase information and has an intersection with the rotation axis of the galvanometer;

[0087] Figure 4 is the flowchart of the steps for establishing the galvanometer scanning model in the embodiment of the present invention. As Figure 4 shown, the step S2 includes the following steps:

[0088] Step S201: Obtain the normal vector of the incident light plane pre-established in the camera coordinate system and the intersection coordinates (x0, y0, z0) between the incident light plane and the rotation axis of the galvanometer;

[0089] As Figure 5 shown, the camera coordinate system O C -X C Y C Z C takes the lens optical center as the origin O C , and the directions parallel to the rows and columns of the pixels are the X C axis and the Y C axis directions respectively. The Z axis is determined according to the right-hand coordinate system ruleC A three-dimensional rectangular coordinate system established in the axial direction.

[0090] Similarly, the normal vector of the incident light plane can be expressed as:

[0091]

[0092] where β is the vector in the Z of the camera coordinate system C O C Y C The included angle between the projection of the plane and the Z-axis, and γ is the vector in the X of the camera coordinate system C O C Y C The included angle between the projection of the plane and the X-axis.

[0093] Due to the existence of the galvanometer installation tolerance, the ideal situation where the incident light ray is completely coincident with the rotation axis of the galvanometer does not exist, that is, the incident light ray cannot be completely coincident with the rotation axis of the galvanometer, and the incident light ray usually deviates and inclines from the rotation axis of the galvanometer. Therefore, it can be assumed that the incident light plane of the light ray must have an intersection point with the rotation axis of the galvanometer, and the intersection point coordinates are set as (x0, y0, z0);

[0094] Step S202: Obtain a temporary coordinate system pre-established on the initial plane of the galvanometer, generate the normal vector of the galvanometer plane based on the temporary coordinate system, and generate the expression of the normal vector of the galvanometer plane in the camera coordinate system according to the rotation matrix between the temporary coordinate system and the camera coordinate system

[0095] As Figure 5 shown, in the temporary coordinate system O G -X G Y G Z G The rotation axis of the galvanometer is used as the Z G axis, the unit vector perpendicular to the Z G axis on the initial galvanometer plane is used as the X G axis, and the intersection point of the X G axis and the Z G axis on the initial galvanometer plane is used as the origin O G , and the unit vector perpendicular to the Z G axis and the X G axis is used as the Y G axis.

[0096] As Figure 6 shown, generating the expression of the normal vector of the galvanometer plane in the camera coordinate system includes the following steps:

[0097] Step M1: Obtain the rotation axis vector of the galvanometer in the camera coordinate system and the normal vector of the galvanometer plane in the temporary coordinate system;

[0098] In the camera coordinate system, the rotation axis vector of the galvanometer can be expressed as:

[0099]

[0100] where θ is the angle between the projection of the vector on the Z C O C Y C plane in the camera coordinate system and the Z C axis, and φ is the angle between the projection of the vector on the X C O C Y C plane in the camera coordinate system and the Z C axis.

[0101] Let the rotation angle of the galvanometer be α. In this temporary coordinate system, the normal vector of the galvanometer plane can be expressed as

[0102] Step M2: Generate a rotation matrix from the temporary coordinate system to the camera coordinate system according to the rotation axis vector of the galvanometer and the normal vector of the galvanometer plane;

[0103] The rotation matrix R from the temporary coordinate system to the camera coordinate system can be expressed as:

[0104]

[0105] Step M3: Convert the normal vector of the galvanometer plane in the temporary coordinate system to the normal vector of the galvanometer plane in the camera coordinate system according to the rotation matrix

[0106] Then the normal vector of the galvanometer plane in the camera coordinate system is:

[0107]

[0108] Step S203: Generate the normal vector of the reflected light plane according to the normal vector of the incident light plane and the normal vector of the galvanometer plane The normal vector of the galvanometer plane is associated with the deflection angle;

[0109] Then in the camera coordinate system, the normal vector of the reflected light plane​ It can be expressed as:

[0110]

[0111] The normal vector of the reflected light plane, denotes the vector and the vector the cosine value of the included angle therebetween.

[0112] Step S204: According to the normal vector of the reflected light plane and the intersection point coordinates (x0, y0, z0), establish the equation of the reflected light plane, and establish a galvanometer scanning model according to the equation of the reflected light plane.

[0113] In this way, in the known camera coordinate system, knowing the normal vector of the reflected light plane and a point (x0, y0, z0) on its plane, the equation π of the reflected light plane o can be expressed as:

[0114] π o : Ax c + By c + Cz c + D = 0

[0115] where A, B, C, and D are all functions of variables x0, y0, z0, α, β, γ, φ.

[0116] α = α0 + k Φ2α Φ(t),

[0117] Therefore, the function f of the reflected light plane at any angle can be expressed as:

[0118] f(x0, y0, z0, α0, k Φ2α , β, γ, φ) = 0;

[0119] According to the function f of the reflected light plane, the target equation f of the galvanometer scanning model to be optimized can be generated:

[0120] F = ∑∥f(x0, y0, z0, α0, k Φ2α , β, γ, φ)∥ 2

[0121] Step S3: Obtain the three-dimensional coordinates (x C , y C , z C ) and the corresponding phase value Φ of multiple points on the calibration plate image. According to the three-dimensional coordinates (x C , y C , z C)The corresponding phase value Φ optimizes and solves the galvanometer scanning model to generate model parameters;

[0122] Figure 7 This is the flowchart of the steps for generating model parameters in the embodiments of the present invention. As Figure 7 shown, the step S3 includes the following steps:

[0123] Step S301: Obtain a pre-established camera model, where the camera model includes the relationship between the pixel coordinate system, the camera coordinate system, and the world coordinate system;

[0124] In the embodiments of the present invention, the camera is regarded as a pinhole imaging model. Multiple calibration plate images with different poses are taken in the camera's field of view. Using the Zhang Zhengyou calibration method, the intrinsic internal parameters of the camera and lens distortion, as well as the rotation and translation matrix from the world coordinate system where the calibration plate 4 is located to the camera coordinate system, are obtained;

[0125]

[0126] In the formula, is the coordinate in the pixel coordinate system, is the coordinate in the camera coordinate system, is the coordinate in the world coordinate system, (u0, v0) is the principal point coordinate, f x , f y is the scale factor on the u and v axes of the image, R is the rotation matrix from the world coordinate system to the camera coordinate system; T is the translation matrix from the world coordinate system to the camera coordinate system, is a 1×3 zero matrix.

[0127] As Figure 5 shown, the pixel coordinate system O P -uv is a two-dimensional rectangular coordinate system established with the upper left corner of the image as the origin O P , and the row and column directions of the pixels are the u and v coordinate axis directions respectively;

[0128] The world coordinate system O W -X W Y W Z W is a three-dimensional rectangular coordinate system established with the upper left corner point of the checkerboard calibration plate 4 as the origin O W , and the row and column directions of the checkerboard are the X W axis and Y W axis directions respectively, and the Z W axis direction is determined according to the right-hand system rule.

[0129] Step S302: Obtain multiple calibration plate images, and based on the camera model, determine the three-dimensional coordinates of the object points corresponding to the image points on the calibration plate images in the camera coordinate system;

[0130] Since the Z-axis coordinate of any point on the calibration board 4 is equal to zero.

[0131] Z W = 0

[0132] Therefore, the camera model can be rewritten as

[0133]

[0134] where T is the product of the camera's internal and external parameters;

[0135]

[0136] Therefore, according to the three-dimensional coordinates of the object points corresponding to the image points on the calibration board image in the camera coordinate system.

[0137] Step S303: Obtain the phase-shift encoded image corresponding to each calibration board image, decode the phase-shift encoded image, generate the phase value Φ of each image point in the calibration board image, and determine the correspondence between the three-dimensional coordinates (x C , y C , z C ) of multiple object points on the calibration board 4 and the phase value Φ;

[0138] In the embodiment of the present invention, when the camera calibrates and shoots the calibration board 4 in different poses, the calibration board pictures projected with the phase-shift encoded pattern under each galvanometer pose are collected at the same time, that is, the phase-shift encoded images are collected at the same time. The three-dimensional coordinates of the corresponding object points on the calibration board 4 in the camera coordinate system can be obtained through the pixel coordinates on the calibration board image.

[0139] Because each calibration board pose has a set of phase-shift encoded images, decoding the phase-shift encoded images can obtain the phase value Φ of each image point in the calibration board image, thus establishing the connection between the three-dimensional coordinates (x C , y C , z C ) of all points on the calibration board 4 and the phase value Φ.

[0140] Step S304: Optimally solve the galvanometer scanning model according to the phase value Φ corresponding to the three-dimensional coordinates (x C , y C , z C ) of multiple points to generate model parameters.

[0141] The galvanometer scanning model can also be expressed as:

[0142]

[0143] d i = A(Φ)x C + B(Φ)y C + C(Φ)zC +D(Φ)

[0144] wherein A(Φ), B(Φ), C(Φ), D(Φ) are parametric models containing x0, y0, z0, α, k Φ2α , β, γ, φ, and m is the number of points taken on the calibration plate.

[0145] Based on the phase values Φ corresponding to the three-dimensional coordinates (x C , y C , z C ), the galvanometer scanning model can be optimized and solved.

[0146] The optimization and solution method can adopt non-linear optimization. When the minimum value of d ij is obtained, the following parametric model is determined:

[0147] x0, y0, z0, α, k Φ2α , β, γ, φ

[0148] In the embodiments of the present invention, in order to avoid too many data points, the (x C , y C , z C , Φ) of all calibration plate corner points are obtained. Since the corner points are sub-pixel coordinates, the corresponding phase values are obtained by bilinear interpolation.

[0149] Step S4: Determine the function associated with the reflected light plane and the phase value Φ according to the model parameters, and determine the coordinates of the object point according to the function associated with the reflected light plane and the phase value Φ and the straight line formed by the connection line between the image point in the camera and the camera optical center point.

[0150] Figure 8 This is the flowchart of the steps for determining the coordinates of the object point in the embodiments of the present invention. As Figure 8 shown, the step S4 includes the following steps:

[0151] Step S401: Determine the galvanometer scanning model according to the model parameters, that is, determine the function associated with the reflected light plane and the phase value Φ;

[0152] Since the equation f of the galvanometer scanning model is:

[0153] d ij = A(Φ)x C + B(Φ)y C + C(Φ)z C + D(Φ)

[0154] Among them, A, B, C, and D are all functions of x0, y0, z0, α, β, γ, and φ are variables. Therefore, knowing the model parameters can determine the equation f of the reflected light plane.

[0155] Step S402: Obtain a target line, which is generated according to the connection line between the image point and the camera optical center point in the camera. Determine the light intensity of the image point according to the pixel value of the image point, and determine the corresponding phase value Φ of the image point according to the multi-step phase shift relationship formula;

[0156] Since the galvanometer scanning model is a function of the phase value Φ, knowing the phase value Φ can determine the function of the reflected light plane determined by the phase value Φ.

[0157] Step S403: Determine the reflected light plane according to the phase value Φ, so that the intersection point of the target line and the reflected light plane can be determined as the object point on the target, and the corresponding object point coordinates are generated.

[0158] The projection of the reflected light plane equation on the target is a straight line, and the three-dimensional coordinates of the object point can be determined through the intersection point of this straight line and the target line.

[0159] Figure 9 It is a schematic diagram of the working principle of the galvanometer scanning model calibration in the embodiment of the present invention. Figure 10 It is a schematic diagram of the control logic of the galvanometer scanning model in the embodiment of the present invention. As Figure 9 、 Figure 10 shown, the PC acts as the host computer, communicates with the FPGA through the serial port, sends instructions to the FPGA, and synchronously sends signals to the line laser, galvanometer 3, and camera through the FPGA. The line laser emits light only when the galvanometer deflects in one direction. Because the period of the galvanometer 3 during scanning is greater than the laser flashing period, light and dark stripes will be formed on the surface of the object to be measured. The camera takes pictures starting from the rising edge of the level, and sets a reasonable exposure time to cover the process of the galvanometer deflecting in one direction to complete a single photo-taking process. The FPGA will send a series of continuous signals to control the shooting of multiple photos with different stripe structures to complete one imaging.

[0160] Figure 11 It is a schematic diagram of the modules of the calibration system of the galvanometer scanning device in the embodiment of the present invention. As Figure 11 shown, the calibration system of the galvanometer scanning device provided by the present invention includes the following modules:

[0161] The correlation information module is used to obtain the correlation relationship between the phase information and the deflection angle. The phase information is the phase of the light intensity of the incident light projected by the light source changing with time, and the deflection angle is the deflection angle of the galvanometer changing with time;

[0162] The model calling module is used to obtain a pre-established galvanometer scanning model, which is established based on the reflected light plane formed by the galvanometer reflecting the reflected light. The reflected light plane is associated with the phase information and has an intersection point with the rotation axis of the galvanometer;

[0163] The model optimization module is used to obtain the three-dimensional coordinates (x C , y C , z C ) and the corresponding phase value Φ of multiple points on the calibration plate image. According to the phase value Φ corresponding to the three-dimensional coordinates (x C , y C , z C ) of multiple points, the galvanometer scanning model is optimized and solved to generate model parameters;

[0164] The object point determination module is used to determine the function associated with the reflected light plane and the phase value Φ according to the model parameters, and determine the coordinates of the object point according to the function associated with the reflected light plane and the phase value Φ and the straight line formed by the connection line between the image point in the camera and the camera optical center point.

[0165] In an embodiment of the present invention, a calibration device for a galvanometer scanning device is further provided, including a processor and a memory. The memory stores executable instructions of the processor. Among them, the processor is configured to execute the calibration method steps of the galvanometer scanning device by executing the executable instructions.

[0166] As above, in this embodiment, a galvanometer scanning model is established based on the fact that the reflected light plane has an intersection point with the rotation axis of the galvanometer, and model parameter calculation is performed to generate a function associated with the reflected light plane and the phase value Φ. Then, for an image point, a target straight line can be determined by connecting it with the camera optical center point and the reflected light plane corresponding to the phase value. The three-dimensional coordinates of the object point on the target are determined according to the intersection point of the target straight line and the reflected light plane, realizing the rapid calculation of the three-dimensional shape information of the object to be measured. Compared with the calibration method based on the complete coaxiality of the rotation axis and the galvanometer, it avoids the calculation error caused by the non-coaxiality of the rotation axis and the galvanometer during installation.

[0167] Those skilled in the art can understand that various aspects of the present invention can be implemented as a system, a method, or a program product. Therefore, various aspects of the present invention can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which can be collectively referred to as "circuit", "module", or "platform" here.

[0168] Figure 12 is a schematic structural diagram of a calibration device for a galvanometer scanning device in an embodiment of the present invention. The following refers to Figure 12 to describe the electronic device 600 according to this embodiment of the present invention. Figure 12The displayed electronic device 600 is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present invention.

[0169] As Figure 12 shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0170] Among them, the storage unit stores program code, and the program code can be executed by the processing unit 610, so that the processing unit 610 executes the steps according to various exemplary embodiments of the present invention described in the calibration method part of the galvanometer scanning device in the above description of this specification. For example, the processing unit 610 can execute as Figure 1 shown in

[0171] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 6201 and / or a cache storage unit 6202, and may further include a read-only storage unit (ROM) 6203.

[0172] The storage unit 620 may also include a program / utilities 6204 having a set (at least one) of program modules 6205. Such program modules 6205 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment.

[0173] The bus 630 may represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any bus structure in a variety of bus structures.

[0174] The electronic device 600 can also communicate with one or more external devices 700 (such as a keyboard, a pointing device, a Bluetooth device, a camera, a depth camera, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (such as a router, a modem, etc.). Such communication can be carried out through an input / output (I / O) interface 650. And, the electronic device 600 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 660. The network adapter 660 can communicate with other modules of the electronic device 600 through the bus 630. It should be understood that althoughFigure 12 which is not shown in the figure, other hardware and / or software modules may be used in combination with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0175] Figure 13 FIG. is a schematic structural diagram of the item picking system in an embodiment of the present invention. The calibration device of the galvanometer scanning device can be set as a part of the item picking system. The item picking system provided by the present invention further includes:

[0176] A first unit and a second unit for storing and / or transporting materials;

[0177] A depth camera 100, whose visual scanning area at least covers the first unit for storing or transporting the materials, is used to visually scan the materials, collect depth images of the materials, and generate pose information of the materials according to the depth images;

[0178] A robot unit 400, which is communicatively connected to the depth camera 100, is used to receive the pose information, and grasp the materials in a corresponding grasping pose according to the pose information and then transfer them to the second unit.

[0179] Wherein, the depth camera 100 includes the galvanometer scanning device, and the above-mentioned processor is configured to perform the calibration method steps of the galvanometer scanning device by executing executable instructions to calibrate the depth camera 100, so as to accurately collect depth images of the materials.

[0180] In an embodiment of the present invention, the first unit may be set as a storage unit 200 and a feeding unit 300;

[0181] A storage unit 200 for storing materials;

[0182] A feeding unit 300 for placing and transporting the storage unit 200;

[0183] The depth camera 100, whose visual scanning area at least covers a part of the transportation path of the transported materials, is used to visually scan the transported materials after they enter the visual scanning area, collect depth images of the transported materials, and generate pose information of the materials according to the depth images;

[0184] A robot unit 400, which is communicatively connected to the depth camera 100, is used to receive the pose information, and grasp the materials in a corresponding grasping pose according to the pose information and then transfer them to the second unit.

[0185] The second unit may be set to perform barcode recognition and transport or store the recognized materials.

[0186] In an embodiment of the present invention, a computer-readable storage medium is further provided for storing a program, and the steps of the calibration method of the galvanometer scanning device are implemented when the program is executed. In some possible implementation manners, various aspects of the present invention can also be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above-mentioned calibration method part of the galvanometer scanning device in this specification.

[0187] As shown above, when the program of the computer-readable storage medium of this embodiment is executed, a galvanometer scanning model is established based on the fact that the reflected light plane and the rotation axis of the galvanometer have an intersection point, and model parameter calculation is performed to generate a function in which the reflected light plane is associated with the phase value Φ. Then, for an image point, a target line can be determined by connecting the line with the camera optical center point and the reflected light plane corresponding to the phase value. The three-dimensional coordinates of the target object point on the target are determined according to the intersection point of the target line and the reflected light plane, realizing the rapid calculation of the three-dimensional shape information of the object to be measured. Compared with the calibration method based on the complete coaxiality of the rotation axis and the galvanometer, it avoids the calculation error caused by the non-coaxiality of the rotation axis and the galvanometer during installation.

[0188] Figure 14 It is a schematic structural diagram of the computer-readable storage medium in an embodiment of the present invention. Refer to Figure 14 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described. It can use a portable compact disc read-only memory (CD-ROM) and include program code, and can run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited to this. In this document, the readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, device, or device.

[0189] The program product can adopt any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0190] A computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The readable storage medium may also be any readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0191] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's device, executed as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., by using an Internet service provider to connect through the Internet).

[0192] In an embodiment of the present invention, a galvanometer scanning model is established based on the intersection of the reflected light plane and the rotation axis of the galvanometer, and model parameter calculation is performed to generate a function in which the reflected light plane is associated with the phase value Φ. Then, for an image point, a target line can be determined by connecting the center point of the camera's optical axis, and the reflected light plane corresponding to the phase value can be obtained. The three-dimensional coordinates of the object point on the target are determined according to the intersection point of the target line and the reflected light plane, realizing the rapid calculation of the three-dimensional shape information of the object to be measured. Compared with the calibration method based on the complete coaxiality of the rotation axis and the galvanometer, the calculation error caused by the non-coaxiality during the installation of the rotation axis and the galvanometer is avoided.

[0193] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts among the embodiments can be referred to each other. The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

[0194] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various deformations or modifications within the scope of the claims, which does not affect the essence of the present invention.

Claims

1. A calibration method for a galvanometer scanning device, the galvanometer scanning device comprising a light source, a camera and a galvanometer, characterized in that, Including the following steps: Obtain the correlation relationship between the phase information and the deflection angle, where the phase information is the phase of the light intensity of the incident light projected by the light source changing with time, and the deflection angle is the deflection angle of the galvanometer changing with time; Obtain the pre-established galvanometer scanning model, which is established based on the reflected light plane formed by the galvanometer reflecting the incident light. The reflected light plane is associated with the phase information and has an intersection with the rotation axis of the galvanometer; Obtain the three-dimensional coordinates (x C , y C , z C ) of multiple points on the calibration board image and the corresponding phase values Φ. Optimize and solve the galvanometer scanning model according to the phase values Φ corresponding to the three-dimensional coordinates (x C , y C , z C ) of multiple points to generate model parameters; Determine the function of the reflected light plane associated with the phase value Φ according to the model parameters, and determine the coordinates of the object point according to the function of the reflected light plane associated with the phase value Φ and the straight line formed by the connection line between the image point and the optical center point of the camera in the camera; 2. The calibration method of the galvanometer scanning device according to claim 1, characterized in that When establishing the correlation relationship between the phase information and the deflection angle, it includes the following steps: Obtain the relationship formula I of the projected light intensity n (t), where the relationship formula I n (t) is the relationship formula in which the light intensity of the incident light projected by the light source is associated with time; Obtain the multi-step phase shift relationship formula, which is the relationship formula between the phase Φ(t) and the light intensity of the incident light. Generate the phase-time relationship formula between the phase Φ(t) and time t according to the projected light intensity relationship formula and the multi-step phase shift relationship formula; Obtain the galvanometer-time relationship formula in which the deflection angle of the galvanometer is associated with time, and generate the correlation formula between the deflection angle and the phase according to the galvanometer-time relationship formula and the phase-time relationship formula; 3. The calibration method of the galvanometer scanning device according to claim 1, characterized in that When establishing the galvanometer scanning model, it includes the following steps: Obtain the normal vector of the incident light plane pre-established in the camera coordinate system and the intersection coordinates (x0, y0, z0) between the incident light plane and the rotation axis of the galvanometer; Obtain a temporarily established coordinate system pre-established on the initial plane of the galvanometer, generate a normal vector of the galvanometer plane based on the temporarily established coordinate system, and generate an expression of the normal vector of the galvanometer plane in the camera coordinate system according to the rotation matrix between the temporarily established coordinate system and the camera coordinate system According to the normal vector of the incident light plane and the normal vector of the galvanometer plane generate the normal vector of the reflected light plane The normal vector of the galvanometer plane is associated with the deflection angle; According to the normal vector of the reflected light plane and the intersection point coordinates (x0, y0, z0), establish the equation of the reflected light plane, and establish a galvanometer scanning model according to the equation of the reflected light plane.

4. The calibration method of the galvanometer scanning device according to claim 2, characterized in that, When solving and generating the model parameters, it includes the following steps: Obtain the pre-established camera model, which includes the relationship between the pixel coordinate system, the camera coordinate system, and the world coordinate system; Obtain multiple calibration plate images, and determine the three-dimensional coordinates of the object point corresponding to the image point on the calibration plate image in the camera coordinate system based on the camera model; Obtain the phase-shift encoded image corresponding to each calibration plate image, decode the phase-shift encoded image, generate the phase value Φ of each pixel in the calibration plate image, and determine the correspondence between the three-dimensional coordinates (x C , y C , z C ) of multiple object points on the calibration plate and the phase value Φ; Optimizing and solving the galvanometer scanning model to generate model parameters according to the phase values Φ corresponding to the three-dimensional coordinates (x C , y C , z C ) of multiple points.

5. The calibration method of the galvanometer scanning device according to claim 2, wherein, When determining the coordinates of the object point, it includes the following steps: Determine the galvanometer scanning model according to the model parameters, that is, determine the function of the reflected light plane associated with the phase value Φ; Obtain the target straight line, which is generated according to the connection line between the image point and the optical center point of the camera in the camera. Determine the light intensity of the image point according to the pixel value of the image point, and determine the phase value Φ corresponding to the image point according to the multi-step phase shift relationship formula; Determine the reflected light plane according to the phase value Φ, so that the intersection point of the target straight line and the reflected light plane can be determined as the object point on the target, and the corresponding object point coordinates are generated; 6. The calibration method of the galvanometer scanning device according to claim 3, characterized in that In the temporary coordinate system, the rotation axis of the galvanometer is the Z G axis, and the unit vector perpendicular to the Z G axis on the initial galvanometer plane is the X G axis. Taking the intersection point of the X G axis and the Z G axis on the initial galvanometer plane as the origin O G , and the unit vector perpendicular to the Z G axis and the X G axis is the Y G axis.

7. The calibration method of the galvanometer scanning device according to claim 6, characterized in that Generate the expression of the normal vector of the galvanometer plane in the camera coordinate system when including the following steps: Obtain the rotation axis vector of the galvanometer in the camera coordinate system and the normal vector of the galvanometer plane in the temporary coordinate system; Generate the rotation matrix from the temporary coordinate system to the camera coordinate system according to the rotation axis vector of the galvanometer and the normal vector of the galvanometer plane; Convert the normal vector of the galvanometer plane in the temporary coordinate system to the normal vector of the galvanometer plane in the camera coordinate system according to the rotation matrix 8. A calibration system for a galvanometer scanning device, the galvanometer scanning device including a light source, a camera, and a galvanometer, characterized in that, Including the following modules: The correlation information module is used to obtain the correlation relationship between the phase information and the deflection angle, where the phase information is the phase of the light intensity of the incident light projected by the light source changing with time, and the deflection angle is the deflection angle of the galvanometer changing with time; The model call module is used to obtain the pre-established galvanometer scanning model, which is established based on the reflected light plane formed by the galvanometer reflecting the reflected light. The reflected light plane is associated with the phase information and has an intersection with the rotation axis of the galvanometer; A model optimization module, configured to obtain the three-dimensional coordinates (x C , y C , z C ) of multiple points on the calibration plate image and the corresponding phase value Φ, and optimize and solve the galvanometer scanning model according to the phase value Φ corresponding to the three-dimensional coordinates (x C , y C , z C ) of multiple points to generate model parameters; An object point determination module, configured to determine a function of the reflected light plane associated with the phase value Φ according to the model parameters, and determine the coordinates of the object point according to the function of the reflected light plane associated with the phase value Φ and the straight line formed by the connection line between the image point in the camera and the camera optical center point.

9. A calibration device for a galvanometer scanning device, characterized in that, Comprising: A processor; A memory module, in which executable instructions of the processor are stored; Wherein, the processor is configured to execute the steps of the calibration method of the galvanometer scanning device according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that, When the program is executed, the steps of the calibration method of the galvanometer scanning device according to any one of claims 1 to 7 are implemented.

Images