A method, device and storage medium for transparent object positioning based on monocular color

Abstract

The present invention relates to a transparent object positioning method, device and storage medium based on monocular color, which are applied to the field of visual positioning technology, including: acquiring corner point information of the calibration image at multiple angles by shooting a calibration image of a calibration plate, solving the camera's internal parameters, external parameters and distortion parameters through the world coordinates and pixel coordinates according to the acquired corner point information, that is, the world coordinates and pixel coordinates of the calibration image, acquiring the image of the transparent object by using a monocular camera, correcting the transparent image by using the calculated parameters, and then inversely deducing the world coordinates of the transparent object according to the mapping relationship between the world coordinates and the pixel coordinates according to the pixel coordinates of the transparent image acquired by shooting, thereby realizing accurate positioning of the transparent object by using a monocular camera, solving the problem in the prior art that a method based on a depth monocular camera and a method based on binocular vision cannot accurately acquire depth information and the positioning is inaccurate.

Description

Technical Field

[0001] The present invention relates to the field of visual positioning technology, and in particular to a transparent object positioning method, device and storage medium based on monocular color. Background Art

[0002] With its own characteristics, vision technology has changed the way robots are used in industrial production. By using image processing technology in machine vision, targets can be identified and located even without the need for high-precision fixtures, which greatly improves the flexibility of industrial robots during use and expands the application scope and practical application value of industrial robots. The current mainstream positioning detection methods are divided into methods based on depth monocular cameras and methods based on binocular vision, but both have defects in practical applications. Depth information cannot be captured for transparent objects, which greatly affects their application in industry.

[0003] Object recognition and pose estimation based on a deep monocular camera first calibrates the monocular camera, then obtains the final point cloud map through the mapping relationship between the depth map and the point cloud map, and finally performs preprocessing according to the actual scene. This method can accurately and reliably obtain the three-dimensional information of the object.

[0004] The principle of mainstream depth cameras such as KinectV2 and Realsense is to estimate depth based on infrared sensors. The infrared projector actively projects the near-infrared spectrum, which forms random reflection spots after irradiating rough objects or penetrating frosted glass, which can then be collected by the infrared monocular camera. By analyzing the infrared image, the depth map of the human body or object within the visible range is calculated. However, for the application scenarios of the present invention, the depth information of transparent objects cannot be correctly obtained, so positioning is impossible;

[0005] As for binocular and monocular cameras, they are more expensive than monocular cameras and have difficulties in feature point matching algorithms, which to a large extent restricts the application of binocular stereo vision in the field of industrial robots. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a transparent object positioning method, device and storage medium based on monocular color to solve the problem in the prior art that depth information cannot be captured and positioning is inaccurate for transparent objects.

[0007] According to a first aspect of an embodiment of the present invention, a method for locating a transparent object based on monocular color is provided, comprising:

[0008] Select multiple calibration images of the checkerboard calibration plate taken at any angle, and calculate the world coordinates and pixel coordinates of each checkerboard corner point on all calibration images;

[0009] According to the world coordinates and pixel coordinates of each checkerboard corner point on the calibration image, the distorted intrinsic parameter matrix, extrinsic parameter matrix and distortion parameters are solved;

[0010] Solve the distortion-free internal parameter matrix based on the solved distorted internal parameter matrix and distortion parameters;

[0011] The image of the transparent object on the fixed plane is captured by a monocular camera, and the image of the transparent object is corrected using the distorted intrinsic parameter matrix and distortion parameters;

[0012] In the corrected image of the transparent object, the pixel coordinates of the transparent object are obtained, and the world coordinates of the transparent object are inversely deduced based on the mapping relationship between the world coordinate system and the pixel coordinate system, as well as the solved distortion-free intrinsic parameter matrix and the extrinsic parameter matrix, and the transparent object is positioned through the world coordinates.

[0013] Preferably,

[0014] The method of capturing an image of a transparent object on a fixed plane by using a monocular camera comprises:

[0015] The images of transparent objects lying down at various angles are captured by a fixed monocular camera to form a data set;

[0016] The brightness of each image in the data set is adjusted up or down based on the original image, each image in the data set is rotated clockwise and counterclockwise by a certain angle, and the adjusted images are added to the data set to expand the data set;

[0017] The data set is trained using an object detection algorithm to obtain a detection model, which can identify transparent objects in a fallen state.

[0018] Preferably,

[0019] The mapping relationship from the world coordinate system to the pixel coordinate system includes:

[0020] The world coordinate system is transformed into a camera coordinate system by rigid body transformation, the camera coordinate system is transformed into an image coordinate system by perspective projection, and the image coordinate system is transformed into a pixel coordinate system by affine transformation;

[0021] The rigid body transformation includes: multiplying the world coordinate system by a rotation matrix and then adding a translation matrix to obtain a camera coordinate system;

[0022] The perspective projection includes: converting the camera coordinate system into a two-dimensional coordinate system of the plane, namely, the image coordinate system, according to the relationship between the focal length, the object distance and the phase distance of the lens imaging under the pinhole model;

[0023] The affine transformation includes: in a two-dimensional plane coordinate system, converting the coordinate origin of the image coordinate system into the midpoint of the pixel coordinate system, thereby realizing the conversion from the image coordinate system to the pixel coordinate system.

[0024] Preferably,

[0025] The distorted internal parameter matrix includes:

[0026] The initial intrinsic parameter matrix is ​​obtained by combining the mapping matrix from the monocular camera coordinate system to the image coordinate system and the mapping matrix from the image coordinate system to the pixel coordinate system;

[0027] Set the unknown number as the distortion parameter of the monocular camera, and determine the radial distortion formula and the tangential distortion formula according to the distortion parameter and the coordinate point of the image coordinate system;

[0028] Combine the radial distortion formula and the tangential distortion formula to obtain the coordinate point of the distorted image coordinate system projected into the pixel coordinate system;

[0029] Substitute the coordinate points of the distorted image coordinate system projected into the pixel coordinate system into the initial intrinsic parameter matrix to obtain the distorted intrinsic parameter matrix;

[0030] The external parameter matrix is ​​obtained according to the mapping matrix from the world coordinate system to the camera coordinate system.

[0031] Preferably,

[0032] The method of solving the distorted intrinsic parameter matrix, extrinsic parameter matrix and distortion parameters according to the world coordinates and pixel coordinates of each checkerboard corner point on the calibration image includes:

[0033] The single-point distorted camera imaging model is obtained by combining the distorted internal parameter matrix and the external parameter matrix;

[0034] Input the world coordinates and pixel coordinates of each chessboard corner point into the single-point distorted camera imaging model;

[0035] Initial values ​​are set for the distorted intrinsic parameter matrix, extrinsic parameter matrix and distortion parameters, and the reprojection error is gradually minimized through a linear optimization method to minimize the initial value error. The initial value after the error is minimized is used as the solved distorted intrinsic parameter matrix, extrinsic parameter matrix and distortion parameters.

[0036] Preferably,

[0037] The step of obtaining a distortion-free internal parameter matrix according to the solved distorted internal parameter matrix and distortion parameters includes:

[0038] According to the solved distorted intrinsic parameter matrix and the mapping relationship between the coordinate points of the distorted image coordinate system and the coordinate points of the pixel coordinate system, the distorted image coordinate points are solved;

[0039] Since the distortion parameters are known, the image coordinate points before distortion are solved according to the mapping relationship between the image coordinate points before distortion and the image coordinate points after distortion;

[0040] Solve the physical size of the horizontal and vertical axes of each pixel in the pixel coordinate system according to the image coordinate points;

[0041] The distortion-free intrinsic parameter matrix is ​​solved according to the physical dimensions of the horizontal and vertical axes of each pixel in the pixel coordinate system.

[0042] According to a second aspect of an embodiment of the present invention, a transparent object positioning device based on monocular color is provided, comprising:

[0043] Calibration module: used to select multiple calibration images of the checkerboard calibration plate taken at any angle, and calculate the world coordinates and pixel coordinates of each checkerboard corner point on all calibration images;

[0044] Distortion solving module: used to solve the distorted intrinsic parameter matrix, extrinsic parameter matrix and distortion parameters according to the world coordinates and pixel coordinates of each checkerboard corner point on the calibration image;

[0045] Distortion-free solution module: used to solve the distortion-free internal parameter matrix according to the solved distorted internal parameter matrix and distortion parameters;

[0046] Correction module: used to capture images of transparent objects on a fixed plane using a monocular camera, and correct the images of transparent objects using the distorted intrinsic parameter matrix and distortion parameters;

[0047] Positioning module: It is used to obtain the pixel coordinates of the transparent object in the corrected image of the transparent object, and reversely infer the world coordinates of the transparent object based on the mapping relationship between the world coordinate system and the pixel coordinate system, as well as the solved distortion-free intrinsic parameter matrix and extrinsic parameter matrix, so as to realize the positioning of the transparent object through the world coordinates.

[0048] According to a third aspect of an embodiment of the present invention, a storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a main controller, each step in the above method is implemented.

[0049] The technical solution provided by the embodiments of the present invention may have the following beneficial effects:

[0050] The present application obtains corner point information of the calibration image from multiple angles by shooting a calibration image of a calibration plate, and solves the camera's intrinsic parameters, extrinsic parameters and distortion parameters through the world coordinates and pixel coordinates based on the obtained corner point information, that is, the world coordinates and pixel coordinates of the calibration image. The image of the transparent object is obtained by using a monocular camera, and the transparent image is corrected through the calculated parameters. Then, the world coordinates of the transparent object are inferred based on the mapping relationship between the world coordinates and the pixel coordinates according to the pixel coordinates of the transparent image obtained by shooting, thereby achieving precise positioning of the transparent object through the monocular camera, solving the problem in the prior art that the method based on the depth monocular camera and the method based on the binocular vision cannot accurately obtain the depth information and the positioning is inaccurate.

[0051] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0053] Figure 1 is a flow chart of a method for locating a transparent object based on monocular color according to an exemplary embodiment;

[0054] Figure 2 is a schematic diagram of a camera coordinate transformation principle according to an exemplary embodiment;

[0055] Figure 3 is a schematic diagram showing the principle of perspective projection according to another exemplary embodiment;

[0056] Figure 4 is a schematic diagram showing the principle of affine transformation according to another exemplary embodiment;

[0057] Figure 5 is a schematic diagram showing the principle of use of a device according to another exemplary embodiment;

[0058] Figure 6 is a system schematic diagram of a transparent object positioning device based on monocular color according to another exemplary embodiment;

[0059] In the attached figure: 1- calibration module, 2- distortion solving module, 3- distortion-free solving module, 4- correction module, 5- positioning module. DETAILED DESCRIPTION

[0060] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0061] Embodiment 1

[0062] Figure 1 is a flow chart of a method for locating a transparent object based on monocular color according to an exemplary embodiment. Figure 1 As shown, the method includes:

[0063] S1, select multiple calibration images of the checkerboard calibration plate taken at any angle, and calculate the world coordinates and pixel coordinates of each checkerboard corner point on all calibration images;

[0064] S2, solving the distorted intrinsic parameter matrix, extrinsic parameter matrix and distortion parameters according to the world coordinates and pixel coordinates of each checkerboard corner point on the calibration image;

[0065] S3, solving the distortion-free internal parameter matrix according to the solved distorted internal parameter matrix and the distortion parameters;

[0066] S4, using a monocular camera to capture an image of a transparent object on a fixed plane, and using a distorted intrinsic parameter matrix and distortion parameters to correct the image of the transparent object;

[0067] S5, in the corrected image of the transparent object, the pixel coordinates of the transparent object are obtained, and the world coordinates of the transparent object are inversely deduced according to the mapping relationship between the world coordinate system and the pixel coordinate system, as well as the solved distortion-free intrinsic parameter matrix and the extrinsic parameter matrix, so as to realize the positioning of the transparent object through the world coordinates;

[0068] It is understandable that a controlled indoor environment is usually required before acquiring image data: a monocular camera with a fixed focal length is placed at a distance of about 1.9 m from the ground; a transparent bottle conveyor plate is placed at a distance of 0.2 m to 0.8 m in front of the monocular camera, as shown in the attached figure. Figure 5As shown, the calibration data set is photographed at various angles, and the calibration result of any one of the calibration images is used as the three-dimensional coordinate reference system to extract the corner point information of the calibration image. Since it is a calibration plate, the world coordinates and pixel coordinates can be obtained at the same time. According to the mapping conversion relationship between the world coordinate system and the pixel coordinate system, the initial intrinsic parameter matrix and the extrinsic parameter matrix of the monocular camera can be determined, and the distortion parameters are set. According to the distortion parameters of the monocular camera, the distorted coordinates in the pixel coordinate system are determined, and the distorted coordinates are substituted into the initial intrinsic parameter matrix to obtain the distorted intrinsic parameter matrix. According to the world coordinates and pixel coordinates of each corner point on the calibration image, the distorted intrinsic parameter matrix, the extrinsic parameter matrix and the distortion parameters are solved; according to the solved distorted intrinsic parameter matrix and distortion parameters, the distortion-free intrinsic parameter matrix is ​​obtained, and the transparent object is photographed by the monocular camera. The transparent object is imaged by a monocular camera, and the image is corrected by a distortion intrinsic parameter matrix and distortion parameters to obtain the pixel coordinates of the transparent object. The world coordinates of the transparent object are inversely deduced according to the mapping relationship between the world coordinate system and the pixel coordinate system, as well as the solved distortion-free intrinsic parameter matrix and the extrinsic parameter matrix, and the transparent object is positioned through the world coordinates. The present application uses a monocular camera to obtain the image of the transparent object, and corrects the transparent image by the calculated parameters, and then according to the pixel coordinates of the transparent image obtained by shooting, the world coordinates of the transparent object are inversely deduced according to the mapping relationship between the world coordinate system coordinates and the pixel coordinates, thereby realizing accurate positioning of the transparent object through the monocular camera, solving the problem in the prior art that the method based on the depth monocular camera and the method based on the binocular vision cannot accurately obtain the depth information and the positioning is inaccurate.

[0069] Preferably,

[0070] The method of taking an image of a transparent object by a monocular camera comprises:

[0071] The images of transparent objects lying down at various angles are captured by a fixed monocular camera to form a data set;

[0072] The brightness of each image in the data set is adjusted up or down based on the original image, each image in the data set is rotated clockwise and counterclockwise by a certain angle, and the adjusted images are added to the data set to expand the data set;

[0073] The detection model is obtained by training the data set through the object algorithm. The detection model can identify transparent objects in a fallen state.

[0074] It is understandable that the object detection algorithm adopts the YOLO (You Only Look Once) series of algorithms. The YOLOv5 algorithm is a typical representative of the current target detection algorithm. Its main idea is to convert the target detection problem into a regression problem. The YOLOv5m model of the YOLOv5 algorithm is selected. It is well compatible with the relationship between the scale of the algorithm network and the detection accuracy. Before using YOLO for detection, it is necessary to collect a data set of transparent objects in a lying state. Under a fixed camera perspective, transparent objects lying at various angles are photographed. In order to further enhance the robustness of the data set, the collected data set is expanded. First, to avoid the influence of lighting, the brightness of each image is adjusted to 0.6 times and 1.4 times of the original image respectively; secondly, to simulate the falling of transparent objects more, each image is rotated 45 degrees clockwise and 45 degrees counterclockwise respectively, and the dataset is trained using YOLO to obtain the detection model. Then, the distortion-corrected monocular camera is turned on to detect transparent objects in the field of view. If a falling transparent object is detected, it will be displayed on the terminal, and the two-dimensional pixel coordinates of the center point of the BoundingBox will be calculated.

[0075] Preferably,

[0076] The mapping relationship from the world coordinate system to the pixel coordinate system includes:

[0077] The world coordinate system is transformed into a camera coordinate system by rigid body transformation, the camera coordinate system is transformed into an image coordinate system by perspective projection, and the image coordinate system is transformed into a pixel coordinate system by affine transformation;

[0078] The rigid body transformation includes: multiplying the world coordinate system by a rotation matrix and then adding a translation matrix to obtain a camera coordinate system;

[0079] The perspective projection includes: converting the camera coordinate system into a two-dimensional coordinate system of the plane, namely, the image coordinate system, according to the relationship between the focal length, the object distance and the phase distance of the lens imaging under the pinhole model;

[0080] The affine transformation includes: in a two-dimensional plane coordinate system, converting the coordinate origin of the image coordinate system into the midpoint of the pixel coordinate system to achieve conversion from the image coordinate system to the pixel coordinate system;

[0081] It is understandable that if Figure 2 As shown, there is a fixed mapping relationship between the world coordinate system and the pixel coordinate system. For example, the world coordinate system can be converted to the camera coordinate system through rigid body transformation, as shown below: The rigid body transformation process is a point in the world coordinate system to a point in the camera coordinate system, which can be described by a rotation matrix R and a translation matrix T. Then there is the following rigid body transformation formula: (1)

[0083]

[0084] In the formula, (U, V, W) represents the coordinate point in the world coordinate system, (X, Y, Z) represents the coordinate point in the camera coordinate system, R is a 3x3 rotation matrix (orthogonal rotation matrix), T is a three-dimensional translation vector, which is converted into homogeneous coordinate form: (2)

[0086]

[0087] The camera coordinate system can be transformed into the image coordinate system through perspective projection. According to the relationship between the focal length of the lens imaging, the object distance and the distance under the pinhole model, as shown in the attached figure: Figure 3 As shown, we can get: (3)

[0089]

[0090] In the formula, f represents the focal length of the monocular camera, x and y represent the coordinate points in the image coordinate system; transforming the relationship in formula (3) into a homogeneous coordinate formula, we get: (4)

[0092]

[0093] The image coordinate system can be converted into the pixel coordinate system through affine transformation. Affine transformation is to translate, scale, flip, rotate and shear objects in a two-dimensional plane. u corresponds to x, v corresponds to y, and the origin of the image coordinate system O′(0, 0) is the midpoint of the pixel coordinate system. Figure 4 As shown, the image coordinate system unit is mm, dx and dy represent the physical size of each pixel on the horizontal axis x and the vertical axis y, that is, the resolution, which represents the actual distance (mm) corresponding to each pixel. dx and dy are shown in formula (5) and formula (6), where u 0 and v 0 are the center positions of the pixel coordinate systems respectively; (5)

[0095] (6)

[0097]

[0098] Therefore, the relationship between the image coordinate system and the pixel coordinate system is shown in formula (7): (7)

[0100]

[0101] In summary, the transformation relationship from the world coordinate system to the pixel coordinate system is shown in formula (8): (8)

[0103]

[0104] Where Z represents the scale factor;

[0105] Preferably,

[0106] Determining the initial intrinsic parameter matrix and extrinsic parameter matrix of the monocular camera according to the mapping relationship from the world coordinate system to the pixel coordinate system includes:

[0107] The initial intrinsic parameter matrix is ​​obtained by combining the mapping matrix from the monocular camera coordinate system to the image coordinate system and the mapping matrix from the image coordinate system to the pixel coordinate system;

[0108] Obtain the external parameter matrix according to the mapping matrix from the world coordinate system to the monocular camera coordinate system;

[0109] It can be understood that the first two matrices of Formula 8 above are combined and calculated, that is, the mapping matrix from the monocular camera coordinate system to the image coordinate system and the mapping matrix from the image coordinate system to the pixel coordinate system are combined to obtain Formula 9: (9)

[0111]

[0112] Where K is the initial internal parameter matrix mentioned above;

[0113] The external parameter matrix depends on the relative position of the camera coordinate system and the world coordinate system, that is, the mapping matrix from the world coordinate system to the monocular camera coordinate system, and the external parameter matrix formula 10 is obtained: (10)

[0115]

[0116] The single-point distortion-free camera imaging model is shown in Equation 11: (11)

[0118]

[0119] Preferably,

[0120] The distorted internal parameter matrix includes:

[0121] The initial intrinsic parameter matrix is ​​obtained by combining the mapping matrix from the monocular camera coordinate system to the image coordinate system and the mapping matrix from the image coordinate system to the pixel coordinate system;

[0122] Set the unknown number as the distortion parameter of the monocular camera, and determine the radial distortion formula and the tangential distortion formula according to the distortion parameter and the coordinate point of the image coordinate system;

[0123] Combine the radial distortion formula and the tangential distortion formula to obtain the coordinate point of the distorted image coordinate system projected into the pixel coordinate system;

[0124] Substitute the coordinate points of the distorted image coordinate system projected into the pixel coordinate system into the initial intrinsic parameter matrix to obtain the distorted intrinsic parameter matrix;

[0125] Obtain the external parameter matrix according to the mapping matrix from the world coordinate system to the camera coordinate system;

[0126] It can be understood that in the process of deriving all the above coordinate system formulas, the linear camera model is followed. However, due to reasons such as the lens manufacturing process, the actual camera cannot achieve the linear model, so that the original image acquired by the camera is distorted. The camera distortion occurs in the process of converting the camera coordinate system to the image coordinate system. Therefore, the distorted camera coordinate system is transformed into: world coordinate system->camera coordinate system->(distortion-free) image coordinate system->(distorted) image coordinate system->pixel coordinate system; the distortion includes radial distortion and tangential distortion. For this reason, this application introduces the distortion parameter: k 1 , k 2 , k 3 , p 1 , p 2 ,

[0127] The radial distortion formula is shown in Equation 12: (12)

[0129]

[0130]

[0131] Where (x, y) is the coordinate point in the linear, distortion-free normalized image coordinate system, is the corresponding distorted normalized image coordinate point, r is the distance from the image coordinate of the coordinate point to the image center point o′, that is, r 2 =x 2 +y 2 ;

[0132] The tangential distortion formula is shown in Equation 13: (13)

[0134]

[0135]

[0136] Combining equations (12) and (13), for a point (x, y) in the image coordinate system, the position of this point on the pixel plane can be calculated;

[0137] The coordinate transformation formula of the distorted image is shown in formula (14): (14)

[0139]

[0140]

[0141] The distorted point is projected onto the pixel plane through the intrinsic parameter matrix to obtain the correct position of the point on the pixel plane, as shown in formula (15): (15)

[0143]

[0144] in, Indicates the coordinates of the center of the camera's photosensitive plate after correction in the pixel coordinate system; (16)

[0146] (17)

[0148]

[0149] The single-point distorted camera imaging model is shown in formula (18): (18)

[0151]

[0152] The first term of the product is the camera intrinsic parameter with distortion, the second term is the camera extrinsic parameter, and f is the focal length of the camera with distortion.

[0153] Preferably,

[0154] The method of solving the distorted intrinsic parameter matrix, extrinsic parameter matrix and distortion parameters according to the world coordinates and pixel coordinates of each checkerboard corner point on the calibration image includes:

[0155] The single-point distorted camera imaging model is obtained by combining the distorted internal parameter matrix and the external parameter matrix;

[0156] Input the world coordinates and pixel coordinates of each chessboard corner point into the single-point distorted camera imaging model;

[0157] Set initial values ​​for the distorted internal parameter matrix, external parameter matrix and distortion parameters, gradually minimize the reprojection error through a linear optimization method to minimize the initial value error, and use the initial value after the error is minimized as the solved internal parameter matrix, external parameter matrix and distortion parameters;

[0158] It can be understood that formula (18) is the single-point distorted camera imaging model obtained by combining the distorted intrinsic parameter matrix and the extrinsic parameter matrix. Since the world coordinates (U, V, W) of each chessboard corner point are known, and the distorted pixel coordinates It is known that the world coordinates of each chessboard corner point and the distorted pixel coordinates are substituted into formula (18), and initial values ​​are set for the distorted internal parameter matrix, the external parameter matrix and the distortion parameter. Through the nonlinear optimization method, the reprojection error is gradually minimized to minimize the initial value error. The initial value after the error is minimized is used as the unknown number in the distorted internal parameter matrix and the external parameter matrix to solve the distorted internal parameter matrix and the external parameter matrix. For the distortion parameter, the initial value with the smallest error is the distortion parameter to be solved.

[0159] Preferably,

[0160] The step of obtaining a distortion-free internal parameter matrix according to the solved distorted internal parameter matrix and distortion parameters includes:

[0161] According to the solved distorted intrinsic parameter matrix and the mapping relationship between the coordinate points of the distorted image coordinate system and the coordinate points of the pixel coordinate system, the distorted image coordinate points are solved;

[0162] Since the distortion parameters are known, the image coordinate points before distortion are solved according to the mapping relationship between the image coordinate points before distortion and the image coordinate points after distortion;

[0163] Solve the physical size of the horizontal and vertical axes of each pixel in the pixel coordinate system according to the image coordinate points;

[0164] Solve the distortion-free intrinsic parameter matrix according to the physical size of the horizontal axis and the vertical axis of each pixel in the pixel coordinate system;

[0165] It can be understood that according to the distorted intrinsic matrix and distortion coefficient, the distorted image, video or real-time shooting is corrected, and the corrected pixel point (u, v) corresponds to the pixel point before correction. According to the above formula (15), since the distorted internal parameter matrix has been solved, the distorted image coordinate point can be solved according to formula (15): Since the distortion parameters are known, the image coordinates (x, y) before distortion can be solved according to formula (14). Then, according to formulas (5), (6) and (9), the new camera intrinsic parameter matrix after correction can be calculated, that is, the undistorted intrinsic parameter matrix. The undistorted intrinsic parameter matrix is ​​the initial intrinsic parameter matrix.

[0166] It can be understood that the pixel coordinates of the transparent object are obtained, the world coordinates of the transparent object are inversely deduced according to the mapping relationship between the world coordinate system and the pixel coordinate system, and the positioning of the transparent object is realized through the world coordinates as follows:

[0167] Known: pixel coordinates (u, v) of transparent objects;

[0168] Derive the image coordinates (x, y):

[0169]

[0170] Derive the camera coordinates (X, Y, Z):

[0171] Derive world coordinates (U, V, W):

[0172] In summary:

[0173] This application is verified through the above scheme, and the data shown in the following table is obtained:

[0174]

[0175] Embodiment 2

[0176] Figure 6 : is a system schematic diagram of a transparent object positioning device based on monocular color according to another exemplary embodiment, comprising:

[0177] Calibration module 1: used to select multiple calibration images of the checkerboard calibration plate taken at any angle, and calculate the world coordinates and pixel coordinates of each checkerboard corner point on all calibration images;

[0178] Distortion solving module 2: used to solve the distorted intrinsic parameter matrix, extrinsic parameter matrix and distortion parameters according to the world coordinates and pixel coordinates of each checkerboard corner point on the calibration image;

[0179] Distortion-free solution module 3: used to solve the distortion-free internal parameter matrix according to the solved distorted internal parameter matrix and distortion parameters;

[0180] Correction module 4: used to capture the image of a transparent object on a fixed plane through a monocular camera, and correct the image of the transparent object using the distorted intrinsic parameter matrix and distortion parameters;

[0181] Positioning module 5: used to obtain the pixel coordinates of the transparent object in the corrected image of the transparent object, and reversely infer the world coordinates of the transparent object based on the mapping relationship between the world coordinate system and the pixel coordinate system, as well as the solved distortion-free intrinsic parameter matrix and the extrinsic parameter matrix, so as to realize the positioning of the transparent object through the world coordinates;

[0182] It can be understood that the present application selects multiple calibration images of the checkerboard calibration plate taken at any angle through the calibration module 1, and calculates the world coordinates and pixel coordinates of each checkerboard corner point on all calibration images; the distortion solving module 2 solves the distorted intrinsic parameter matrix, the extrinsic parameter matrix and the distortion parameters according to the world coordinates and pixel coordinates of each checkerboard corner point on the calibration image; the distortion-free solving module 3 solves the distortion-free internal parameter matrix according to the solved distorted internal parameter matrix and the distortion parameters; the correction module 4 uses a monocular camera to shoot an image of a transparent object on a fixed plane, and uses the distorted internal parameter matrix and the distortion parameters to correct the image of the transparent object; the positioning module 5 obtains the corrected transparent object image. Take the pixel coordinates of the transparent object, reverse the world coordinates of the transparent object according to the mapping relationship between the world coordinate system and the pixel coordinate system, and the solved distortion-free intrinsic parameter matrix and the extrinsic parameter matrix, and realize the positioning of the transparent object through the world coordinates; the application uses a monocular camera to obtain the image of the transparent object, and corrects the transparent image through the calculated parameters, and then reverses the world coordinates of the transparent object according to the mapping relationship between the world coordinates and the pixel coordinates of the transparent image obtained by shooting, thereby realizing the precise positioning of the transparent object through the monocular camera, solving the problem in the prior art that the method based on the depth monocular camera and the method based on the binocular vision cannot accurately obtain the depth information and the positioning is inaccurate.

[0183] Embodiment three:

[0184] This embodiment provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a main controller, each step in the above method is implemented;

[0185] It is understandable that the storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0186] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0187] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0188] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0189] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0190] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0191] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0192] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0193] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0194] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A transparent object positioning method based on monocular color, characterized in that: The method comprises: Select multiple calibration images of the checkerboard calibration plate taken at any angle, and calculate the world coordinates and pixel coordinates of each checkerboard corner point on all calibration images; According to the world coordinates and pixel coordinates of each checkerboard corner point on the calibration image, the distorted intrinsic parameter matrix, extrinsic parameter matrix and distortion parameters are solved; Solve the distortion-free internal parameter matrix based on the solved distorted internal parameter matrix and distortion parameters; The step of obtaining a distortion-free internal parameter matrix according to the solved distorted internal parameter matrix and distortion parameters includes: According to the solved distorted intrinsic parameter matrix and the mapping relationship between the coordinate points of the distorted image coordinate system and the coordinate points of the pixel coordinate system, the distorted image coordinate points are solved; Since the distortion parameters are known, the image coordinate points before distortion are solved according to the mapping relationship between the image coordinate points before distortion and the image coordinate points after distortion; Solve the physical size of the horizontal and vertical axes of each pixel in the pixel coordinate system according to the image coordinate points; Solve the distortion-free intrinsic parameter matrix according to the physical size of the horizontal axis and the vertical axis of each pixel in the pixel coordinate system; The image of the transparent object on the fixed plane is captured by a monocular camera, and the image of the transparent object is corrected using the distorted intrinsic parameter matrix and distortion parameters; In the corrected image of the transparent object, the pixel coordinates of the transparent object are obtained, and the world coordinates of the transparent object are inversely deduced based on the mapping relationship between the world coordinate system and the pixel coordinate system, as well as the solved distortion-free intrinsic parameter matrix and the extrinsic parameter matrix, and the transparent object is positioned through the world coordinates.

2. The method according to claim 1, characterized in that The method of capturing an image of a transparent object on a fixed plane by using a monocular camera comprises: The images of transparent objects lying down at various angles are captured by a fixed monocular camera to form a data set; The brightness of each image in the data set is adjusted up or down based on the original image, each image in the data set is rotated clockwise and counterclockwise by a certain angle, and the adjusted images are added to the data set to expand the data set; The data set is trained using an object detection algorithm to obtain a detection model, which can identify transparent objects in a fallen state.

3. The method according to claim 1, characterized in that The mapping relationship from the world coordinate system to the pixel coordinate system includes: The world coordinate system is transformed into a camera coordinate system by rigid body transformation, the camera coordinate system is transformed into an image coordinate system by perspective projection, and the image coordinate system is transformed into a pixel coordinate system by affine transformation; The rigid body transformation includes: multiplying the world coordinate system by a rotation matrix and then adding a translation matrix to obtain a camera coordinate system; The perspective projection includes: converting the camera coordinate system into a two-dimensional coordinate system of the plane, namely, the image coordinate system, according to the relationship between the focal length, the object distance and the phase distance of the lens imaging under the pinhole model; The affine transformation includes: in a two-dimensional plane coordinate system, converting the coordinate origin of the image coordinate system into the midpoint of the pixel coordinate system, thereby realizing the conversion from the image coordinate system to the pixel coordinate system.

4. The method according to claim 3, characterized in that The distorted internal parameter matrix includes: The initial intrinsic parameter matrix is ​​obtained by combining the mapping matrix from the monocular camera coordinate system to the image coordinate system and the mapping matrix from the image coordinate system to the pixel coordinate system; Set the unknown number as the distortion parameter of the monocular camera, and determine the radial distortion formula and the tangential distortion formula according to the distortion parameter and the coordinate point of the image coordinate system; Combine the radial distortion formula and the tangential distortion formula to obtain the coordinate point of the distorted image coordinate system projected into the pixel coordinate system; Substitute the coordinate points of the distorted image coordinate system projected into the pixel coordinate system into the initial intrinsic parameter matrix to obtain the distorted intrinsic parameter matrix; The external parameter matrix is ​​obtained according to the mapping matrix from the world coordinate system to the camera coordinate system.

5. The method according to claim 4, characterized in that The method of solving the distorted intrinsic parameter matrix, extrinsic parameter matrix and distortion parameters according to the world coordinates and pixel coordinates of each checkerboard corner point on the calibration image includes: The single-point distorted camera imaging model is obtained by combining the distorted internal parameter matrix and the external parameter matrix; Input the world coordinates and pixel coordinates of each chessboard corner point into the single-point distorted camera imaging model; Initial values ​​are set for the distorted intrinsic parameter matrix, extrinsic parameter matrix and distortion parameters, and the reprojection error is gradually minimized through a linear optimization method to minimize the initial value error. The initial value after the error is minimized is used as the solved distorted intrinsic parameter matrix, extrinsic parameter matrix and distortion parameters.

6. A transparent object positioning device based on monocular color, characterized in that: The device comprises: Calibration module: used to select multiple calibration images of the checkerboard calibration plate taken at any angle, and calculate the world coordinates and pixel coordinates of each checkerboard corner point on all calibration images; Distortion solving module: used to solve the distorted intrinsic parameter matrix, extrinsic parameter matrix and distortion parameters according to the world coordinates and pixel coordinates of each checkerboard corner point on the calibration image; Distortion-free solution module: used to solve the distortion-free internal parameter matrix according to the solved distorted internal parameter matrix and distortion parameters; The step of obtaining a distortion-free internal parameter matrix according to the solved distorted internal parameter matrix and distortion parameters includes: According to the solved distorted intrinsic parameter matrix and the mapping relationship between the coordinate points of the distorted image coordinate system and the coordinate points of the pixel coordinate system, the distorted image coordinate points are solved; Since the distortion parameters are known, the image coordinate points before distortion are solved according to the mapping relationship between the image coordinate points before distortion and the image coordinate points after distortion; Solve the physical size of the horizontal and vertical axes of each pixel in the pixel coordinate system according to the image coordinate points; Solve the distortion-free intrinsic parameter matrix according to the physical size of the horizontal axis and the vertical axis of each pixel in the pixel coordinate system; Correction module: used to capture images of transparent objects on a fixed plane using a monocular camera, and correct the images of transparent objects using the distorted intrinsic parameter matrix and distortion parameters; Positioning module: It is used to obtain the pixel coordinates of the transparent object in the corrected image of the transparent object, and reversely infer the world coordinates of the transparent object based on the mapping relationship between the world coordinate system and the pixel coordinate system, as well as the solved distortion-free intrinsic parameter matrix and extrinsic parameter matrix, so as to realize the positioning of the transparent object through the world coordinates.

7. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the main controller, each step in the monocular color-based transparent object positioning method as described in any one of claims 1 to 5 is implemented.

Images