Ranging control method based on structured light module
By fitting the modeling parameters with pixel coordinates and distance information, the distance measurement function is constructed, which simplifies the camera calibration process of structured light modules, solves the problems of low calibration efficiency and large error in the existing technology, and realizes efficient distance measurement operate.
Patent Information
- Application Number
- CN202310606044.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-26
- Publication Date
- 2024-11-26
AI Technical Summary
The existing structured light vision system is inefficient in camera calibration and laser plane calibration, affecting the ranging efficiency, and requires a lot of calculations to determine the camera internal parameters, increasing calibration errors.
By using pixel coordinates and distance information to fit modeling parameters, a distance measurement function is constructed, the camera calibration steps are simplified, the calculation amount is reduced, and the distance measurement efficiency is improved. Specific methods include controlling the structured light emitter to emit structured light, collecting the calibration plate image, extracting the center pixel coordinates of the light bar, non-linear fitting to obtain modeling parameters, constructing the structured light distance measurement relationship, and realizing automated calibration.
The calibration efficiency of the structured light module is improved, calibration error is reduced, the calculation of camera internal references is simplified, and the efficiency of ranging operation is improved.
Smart Images

Figure CN119024345A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of structured light ranging and camera calibration, and in particular to a ranging control method based on a structured light module. Background Art
[0002] The structured light vision system is a measurement system that combines a line laser and a monocular camera. For example, a line laser sensor is composed of a camera and a line laser. Before using the structured light vision system, camera calibration and laser plane calibration are required to achieve three-dimensional measurement through imaging principles and laser plane parameters. This calibration method is applicable to most cameras. The line laser projects a line laser onto the surface of the object to be measured and forms a line laser plane. The line laser is reflected into the receiving lens of the camera and forms an image on the photosensitive element of the camera. The position of the image changes with the distance of the target. Since the line laser plane can be approximately regarded as a combination of countless single-point lasers emitted from the laser light source, technicians in related fields will obtain three-dimensional spatial coordinates for the parameters of the light plane where some discrete laser points are located and the camera internal parameters. In this process, they rely on the help of targets such as chessboards. The calibration calculation of the internal parameter matrix is large, and the overall calibration efficiency is relatively low, which affects the implementation of the distance measurement operation of the structured light vision system and reduces the distance measurement efficiency of the structured light vision system that needs to be calibrated. Summary of the invention
[0003] This application discloses a distance measurement control method based on a structured light module. The specific technical solution is as follows: A distance measurement control method based on a structured light module, the structured light module comprising a structured light emitter and a camera; the distance measurement control method comprising: step A, fitting modeling parameters using pixel coordinates and distance information, and constructing a distance measurement function with the modeling parameters as known quantities; step B, controlling the structured light emitter to emit structured light to the surface of the object to be measured and form light strips, and controlling the camera to collect an image of the structured light projected onto the surface of the object to be measured; step C, calculating the light strip center pixel coordinates from the image collected by the structured light module using a light strip center extraction algorithm; and step D, inputting the calculated light strip center pixel coordinates into the distance measurement function, and converting the distance between the object to be measured and the structured light module.
[0004] Combining steps A to D, this embodiment uses pixel coordinates and distance information to fit modeling parameters, reduces camera detection errors, and overcomes the problem that the image information and distance information collected by the structured light module in the calibration device are not linearly related; and after fitting the modeling parameters, a distance measurement function is further constructed to achieve the effect of simplifying the camera calibration steps, speeding up the acquisition of distance information, and improving the efficiency of the robot using the aforementioned structured light module for distance measurement.
[0005] Furthermore, after the light strip is imaged by the camera, it is represented by a plurality of corresponding columns of pixels in the pixel plane of the camera; each column of pixels forms a corresponding measurement angle relative to the optical axis of the camera; each column of pixels is used to identify the pixel position formed by the light spot of the light strip at each measurement angle in the photosensitive element of the camera; each column of pixels fits the corresponding modeling parameters in step A, and the distance measurement function constructed for each column of pixels in step A corresponds to a measurement angle; wherein, the structured light emitted by the structured light emitter can be regarded as composed of a plurality of single-point line lasers to form a structured light surface; within the structured light surface, a single-point line laser forms a corresponding measurement angle relative to the optical axis of the camera. The light strip is the intersection of the structured light surface and the measured surface of the calibration plate. By fitting the corresponding modeling parameters at a finite number of measurement angles, and then converting the distance information of the measured object at a finite number of measurement angles of the camera based on the distance measurement function, some simple contour point distance measurement application scenarios can be met.
[0006] Furthermore, whenever step C calculates the light bar center pixel coordinates at a measurement angle, step D calculates the distance between the measured object and the structured light module at the corresponding measurement angle, representing the distance information at the position of the corresponding measurement angle; wherein, there is at least one light bar center pixel coordinate in a column of pixels, so that when step D uses the distance measurement function for conversion, the light point pixel coordinates are configured to be represented by the light bar center pixel coordinates. Thus, the single-point line laser and camera at each measurement angle can be modeled in turn, N distance measurement functions can be constructed, and N groups of distance information between the measured object and the structured light module can be converted accordingly, so as to achieve the distance between the position points in each direction covered by the light bar in the measured object and the structured light module through step D.
[0007] Further, in step C, the light stripe center extraction algorithm is used to extract the light stripe center line in the image collected by the structured light module, and multiple light stripe center pixel coordinates are obtained to form a center line point set; in the image including the light stripe, there is at least one light stripe center pixel coordinate in a column of pixels; wherein the light stripe center extraction algorithm includes but is not limited to the grayscale centroid extraction algorithm. It is possible to relatively accurately extract the light stripe center pixel coordinates while ensuring the accuracy of the added internal parameters, and use the distance measurement function to speed up the distance measurement operation.
[0008] Furthermore, in step A, the method for fitting modeling parameters using pixel coordinates and distance information includes: step A1, controlling the structured light emitter to emit structured light, projecting it onto the calibration plate to form a light strip; at the same time, controlling the calibration plate to move relative to the structured light module, and the camera collects calibration plate images at different test distances to obtain multiple frames of calibration plate images, and extracts the pixel coordinates of light spots in each column from the same frame of the calibration plate image, and determines the measurement distance corresponding to the same column of the same frame of the calibration plate image; step A2, in the multiple frames of the calibration plate images obtained in step A1, performing nonlinear fitting on the pixel coordinates of light spots in the same column and their corresponding test distances, Obtain modeling parameters corresponding to the same column; step A3, by repeatedly executing step A2 to traverse each column of light spot pixel coordinates and their corresponding test distances in each frame of the calibration plate image obtained in step A1, obtain the modeling parameters corresponding to all columns in turn, and determine to complete the calibration of the modeling parameters of the structured light module; the modeling parameters corresponding to the same column are the coefficients required to construct the structured light ranging relationship under the same column. The structured light ranging relationship is a relationship between the light spot pixel coordinates in the same column and the corresponding test distances constructed based on the pinhole imaging principle, so that the structured light ranging relationship under the same column forms the function to be fitted corresponding to the nonlinear fitting of step A2.
[0009] In step A2, the aforementioned modeling parameters can be calculated by performing nonlinear fitting on the mapping relationship between the pixel coordinates of the light spots in the same column of the multiple frames of the calibration plate images and their test distances, as the optimal parameters required to construct the mapping relationship between the pixel coordinates of the light spots in the same column and their test distances, that is, the coefficients required to construct the structured light ranging relationship. The fitted modeling parameters can replace the traditional camera intrinsic parameters in a direction or a measurement angle relative to the optical axis of the camera.
[0010] Furthermore, before executing the step A1, it includes: adjusting the emission direction of the structured light emitter to be set at an angle with the optical axis direction of the camera, setting the line between the center of the structured light emitter and the center of the camera to be parallel to the measured surface of the calibration plate; setting the emission direction of the structured light emitter to intersect with the calibration plate; wherein the imaging of the reflected light of the structured light collected by the camera in the photosensitive element of the camera can be regarded as a combination of various imaging positions formed in the imaging plane after the multiple single-point line lasers constituting the structured light are reflected by the calibration plate; the imaging of the reflected light of the structured light collected by the camera in the photosensitive element of the camera is represented by the imaging of the light strips in the imaging plane of the camera. On the basis of keeping the relative posture between the camera and the structured light emitter unchanged, the distance information of each position of the projected structured light in the calibration plate can be calculated and restored faster by using the pinhole imaging principle, especially when the structured light emitter is equivalent to being decomposed into multiple single-point line laser emitters, the distance measurement information at multiple angles can be obtained.
[0011] Furthermore, in the step A1, the method of controlling the displacement of the calibration plate relative to the structured light module includes: driving the calibration plate to slide on the slide rail; each time the calibration plate slides a certain distance along the slide rail, the camera collects an image of the calibration plate, the light bar is offset once on the calibration plate, and the camera detects the pixel coordinates of the light spot formed by the light bar in the pixel plane in real time, and records the test distance at the corresponding measurement angle; wherein the relative position between the structured light emitter and the camera is fixed by a clamp. Therefore, the calibration plate calibrates the structured light module under the drive of the motor, and there is no need to specially set up targets such as a checkerboard. The calibration plate can be controlled by the motor to slide along the slide rail to collect the distance information and pixel coordinates required for fitting the corresponding distance measurement relationship under the pinhole imaging model, and automation can be easily realized; thereby, while ensuring the reliability of the calibration parameters or calibration results of the structured light module, the efficiency of the structured light module calibration is improved and the calibration error is reduced.
[0012] Furthermore, in the step A1, during the sliding of the calibration plate along the slide rail, each time the calibration plate slides a preset fixed distance, the camera captures the image of the light bar, the imaging position of the light bar in the imaging plane is offset by a preset height along the height direction, and the pixel coordinates of the light spot at the corresponding measurement angle are offset by a preset longitudinal offset in the opposite direction of the height direction in the pixel plane; wherein the offset formed by the imaging position of the light bar in the imaging plane at the same measurement angle relative to the reference position at the same measurement angle is marked as the light bar offset at the same measurement angle; wherein the reference position is the imaging position of the reflected light of the structured light emitted by the structured light emitter on the measured surface of the measured object farthest from the structured light module in the camera within the detectable area of the structured light module; the measured object includes the calibration plate. At each measurement angle, the reference position is used to represent the offset reference point of the projection position of the light bar in the imaging plane coordinate system, and provides an imaging position offset starting point for the measured object or calibration plate at any distance, so as to facilitate the measurement and description of the displacement of the light bar in the imaging plane or in the photosensitive element from the reference position. Moreover, during the sliding process of the calibration plate, by setting the preset longitudinal offset as the stepping distance, it is convenient to measure the pixel coordinates of the detection light spot from a pixel offset starting point in the pixel plane of the camera.
[0013] Further, in the same column, the method for constructing the structured light ranging relationship includes: setting the line between the center of the structured light emitter and the optical center of the camera as the baseline, and setting the length of the baseline as the baseline distance; setting the distance between the baseline and the calibration plate at the corresponding measurement angle as the test distance d at the corresponding measurement angle; the structured light ranging relationship at the corresponding measurement angle is: Px=k0 / d+k1, forming a relationship between the pixel coordinates of the light point at the orientation represented by a column of pixel points and the test distance; wherein Px is the pixel coordinates of the light point at the corresponding measurement angle; wherein the modeling parameter package corresponding to the same column k0 and k1 are included; at the same measurement angle, the ratio of the baseline distance to the test distance is equal to the ratio of the light bar offset to the focal length of the camera; at the same measurement angle, the sum of the product of the light spot pixel coordinates and the single pixel size and the projection error is equal to the light bar offset; the single pixel size and the projection error are both predetermined; the imaging of the light bar in the imaging plane of the camera is set in the imaging plane coordinate system, wherein the reference position and the light bar offset are both set in the imaging plane coordinate system; the light spot pixel coordinates are set in the pixel coordinate system, and the pixel coordinate system is the coordinate system of the pixel plane of the camera. Reducing the type and number of parameters to be calculated is equivalent to reducing the number of camera intrinsic parameters to be calibrated, simplifying the camera calibration method. Therefore, only the two parameters k0 and k1 in the aforementioned structured light ranging relationship need to be determined to complete the modeling of the structured light module, that is, to establish a model for measuring the distance between the baseline and the object to be measured using structured light, which is equivalent to completing the calibration of the camera, and has simplified the camera calibration method. Then, on the basis of simplifying the camera intrinsic calibration operation, the functional relationship d=k0 / (Px-k1) between the test distance and the light spot pixel coordinates is determined, which is used to calculate the test distance when the light spot pixel coordinates are known as the ranging information of the structured light module.
[0014] Further, in step A, the method of constructing a distance measurement function with modeling parameters as known quantities includes: determining that the structured light distance measurement relationship at the corresponding measurement angle is Px=k0 / d+k1, and when the modeling parameters k0 and k1 at the corresponding measurement angle are obtained through step A3, the light spot pixel coordinates Px at the corresponding measurement angle and the test distance d at the corresponding measurement angle are set as unknown quantities; then Px=k0 / d+k1 is converted to d=k0 / (Px-k1), and the converted d=k0 / (Px-k1) is recorded as the distance measurement function at the corresponding measurement angle. Determine that the construction of the distance measurement function at the corresponding measurement angle is completed. Therefore, whenever step C calculates the center pixel coordinates of the light strip at a measurement angle, step D converts the distance between the object under test and the structured light module at the corresponding measurement angle, representing the distance information at the position of the corresponding measurement angle.
[0015] Further, in step A2, the method for performing nonlinear fitting on the pixel coordinates of light spots in the same column and their corresponding test distances includes: step 21, sequentially traversing the i-th column of the M frames of calibration plate images collected in step A1; wherein the i-th column is a column of pixel positions where the light spots of the light strip at the corresponding measurement angle are projected in the photosensitive element of the camera, and is recorded as the calibration column; step 22, extracting the row pixel position coordinates Pxj of the center of the light strip in the i-th column from the currently traversed frame of calibration plate images, and recording them as the i-th column of the pixel coordinates of light spots in the currently traversed frame of calibration plate images, and determining the test distance dj corresponding to the i-th column of the same frame of calibration plate images; j represents the sorting number of the currently traversed frame of calibration plate images, so that the currently traversed frame of calibration plate images is marked as the j-th frame of calibration plate images; j is a positive integer, j is greater than 0, and j is less than or equal to M; step 23, configuring j=j+1, so that The data of the j+1th calibration plate image is updated to the data of the jth calibration plate image, and step 22 is repeated until the i-th column of the M-frame calibration plate image is traversed, and the pixel coordinates of the light spots in the i-th column of each frame of the calibration plate image are combined into a coordinate set {Px1, Px2, ..., PxM}, and the test distances corresponding to the i-th column in each frame of the calibration plate image are combined into a distance set {d1, d2, ..., dM} or a distance set {1 / d1, 1 / d2, ..., 1 / dM}; step 24, configure the coordinate set of step 23 and the distance set of step 23 as data sets on two coordinate axes, perform nonlinear fitting, obtain the corresponding modeling parameters k0 and modeling parameters k1 of the i-th column, and determine the structured light ranging relationship corresponding to the i-th column; and record the modeling parameters k0 and modeling parameters k1 corresponding to the i-th column as a modeling parameter pair of the i-th column; then execute step A3. The modeling parameters k0 and k1 corresponding to the same column fitted in step 24 satisfy the structured light ranging relationship corresponding to the same column collected by the calibration plate moving along the slide rail, and are further extended to meet the ranging requirements of the structured light module for the object being measured at the corresponding measurement angle, that is, the fitting coefficients required for the ranging model of the structured light module at the corresponding measurement angle.
[0016] Compared with the existing technology, the type and number of camera intrinsic parameters that need to be calculated are reduced, thereby simplifying the calculation of the transformation relationship from the camera coordinate system to the pixel coordinate system and improving the calibration efficiency of the structured light module; the calculation efficiency is also improved when it is subsequently applied to ranging.
[0017] Further, in step A3, i=i+1 is configured so that the pixels in the i+1th column in the jth frame calibration plate image are updated to the pixels in the ith column in the jth frame calibration plate image, and steps 21 to 24 are repeated until the N columns in each frame calibration plate image acquired in step A1 are traversed, and it is determined that after the N columns are nonlinearly fitted, N modeling parameter pairs are cumulatively obtained, and it is determined that the calibration of the modeling parameters of the structured light module is completed; in M frames of calibration plate images, N structured light ranging relationship equations are cumulatively fitted by repeating steps 21 to 24; the N modeling parameter pairs correspond to the N calibrated columns in each frame calibration plate image, and each column corresponds to a measurement angle and a test distance; i represents the column number of the currently traversed column in a frame calibration plate image, i is a positive integer, i is greater than 0, and i is less than or equal to N. Therefore, the technical solution repeatedly executes steps 21 to 24 to complete the calibration of the corresponding modeling parameters column by column in each frame of the calibration plate image, thereby completing the calibration of the structured light module from multiple measurement angles.
[0018] Furthermore, nonlinear fitting includes but is not limited to the least square method to minimize the fitting error. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 It is a flow chart of a distance measurement control method based on a structured light module disclosed in one embodiment of the present invention.
[0020] Figure 2 It is a schematic diagram of the optical path of a single-point line laser disclosed in one embodiment of the present invention.
[0021] Figure 3 It is a schematic diagram of the ranging principle of a structured light emitter and a camera disclosed in an embodiment of the present invention.
[0022] Figure 4 It is a specific flow chart of step A disclosed in one embodiment of the present invention. Implementation
[0023] The technical solutions in the embodiments of the present invention will be described in detail below in conjunction with the drawings in the embodiments of the present invention. To further illustrate the embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in this field should be able to understand other possible implementations and advantages of the present invention.
[0024] As an embodiment, a distance measurement control method based on a structured light module is provided, wherein the structured light module includes a structured light emitter and a camera; Figure 1 As shown, the ranging control method includes the following steps: Step A, using pixel coordinates and distance information to fit modeling parameters, specifically using the structured light module to collect pixel coordinates and distance information on the calibration device for fitting processing, obtaining modeling parameters at each measurement angle, and constructing a ranging function with the modeling parameters as known quantities. In this embodiment, the image collected by the camera is used for ranging, so the ranging function is a variant of the structured light ranging relationship constructed by the structured light module under the pinhole imaging principle, and the ranging function serves as the ranging model of the structured light module.
[0025] Step A generally first determines and uses the pixel coordinates and distance information measured in real time in the calibration device based on the pinhole imaging model to fit the structured light ranging relationship, as well as the coefficients required to construct the structured light ranging relationship, i.e., the modeling parameters. In the specific implementation process, there is a motor, a calibration plate, a slide rail, and a fixture for fixing the sensor in the structured light module to form the calibration device, which is used to calibrate the structured light module under the drive of the motor. Therefore, the structured light module relies on the calibration plate to step along the slide rail to collect each column of pixel coordinates and their corresponding test distances to fit the mathematical relationship between the pixel coordinates of the light point constructed under the pinhole imaging principle (the point projected by the structured light emitter onto the calibration plate or the surface of the object to be measured, and the projection point of a single-point line laser equivalently decomposed from the structured light emitted by the structured light emitter) and the test distance (the distance of the object to be measured). After fitting the corresponding modeling parameters, it can be determined that the calibration of the camera is completed, thereby improving the calibration efficiency. The structured light ranging relationship will be converted into a ranging function with the modeling parameters as known quantities, which can be regarded as a ranging mathematical model for the structured light module at a measurement angle (relative to a direction of the optical axis of the camera) in the placement plane where the slide rail is located, thereby achieving the improvement of the calibration efficiency by relying on the ranging function to accelerate the ranging efficiency.
[0026] Step B, control the structured light emitter to emit structured light to the surface of the object to be measured and form light strips, and control the camera to collect the image of the structured light projected onto the surface of the object to be measured, the image of the surface of the object to be measured includes the image of the light strips; the structured light emitted by the structured light emitter can be a single-point line laser or a multi-line laser matrix. In step B, the structured light module is not set on the slide rail, and it is not necessary to collect the image of the calibration plate. The structured light module can be set on the robot to perform ranging operations in the navigation environment, so it does not rely on the calibration device to calibrate the camera. The camera used to collect the image including the light strips in step B is regarded as a calibrated camera, that is, the structured light module has been calibrated in step A.
[0027] The structured light emitted by the structured light emitter changes the optical path through the camera so that the structured light forms a structured light surface. The structured light surface can be regarded as being decomposed into multiple single-point line lasers according to the measurement angle. The light source of the structured light emitter is only one point, and the optical path is changed by the lens and the wave mirror, so that the emitted structured light forms an effect of an approximate plane, which is equivalent to a structured light surface, such as Figure 3 The structured light surface shown, Figure 3 As shown, the structured light emitter can be regarded as emitting multiple single-point line lasers to form a structured light surface. The intersection of the structured light surface and the surface of the object to be measured is represented by the light strip shown in the figure, and the camera is controlled to collect the original image including the light strip. For example, a straight line laser can emit a laser plane. The laser plane emitted by it can be regarded as composed of multiple single-point line lasers on the basis of determining multiple measurement angles. The straight line laser can emit a laser plane that intersects with the measured surface of the calibration plate to form a laser light strip. The camera will collect a two-dimensional laser stripe image. When using single-point laser triangulation ranging, that is, when using the pinhole imaging model (pinhole model) to establish a mathematical model or range measurement, it is necessary to decompose the structured light into multiple single-point line lasers to form a corresponding optical path of the single-point line laser.
[0028] The structured light module can actively or under the command of an external controller extract height calibration data, the X-axis component or Y-axis component of the light strip point cloud, the X-direction calibration data of the black and white area or the white area in the calibration plate, etc. from the image of the object to be measured, and can be used to convert the row pixel position information of the light strip (or the center line of the light strip), including the light spot pixel coordinates, wherein the light spot pixel coordinates can be regarded as the projection coordinates of the position point on the intersection line of the structured light surface emitted by the structured light emitter and the surface of the object to be measured in the pixel coordinate system of the camera, and support the extraction of the light strip center extraction algorithm disclosed by the prior art, and configure it as the light strip center pixel coordinates.
[0029] Step C: Calculate the light stripe center pixel coordinates from the image captured by the structured light module using a light stripe center extraction algorithm, including the light stripe center pixel coordinates corresponding to each column of pixels or the light stripe center pixel coordinates at each measurement angle, all of which are set in the pixel plane.
[0030] Among them, the center pixel coordinates of the light bar are used to indicate that the structured light emitted by the structured light module is reflected by the surface of the object being measured and projected onto the pixel position in the pixel plane of the structured light module, located at the center position of each column of pixels in the ROI (region of interest) or at the candidate center point.
[0031] Step D, inputting the calculated light strip center pixel coordinates into the ranging function, converting the distance between the measured object and the structured light module, specifically converting the test distance, and setting the converted test distance as the distance between the measured object and the structured light module, wherein the basis for using the ranging function for conversion is that the aforementioned modeling parameters are known parameters, which will not change due to changes in distance within the allowable error range, and the light strip center pixel coordinates are a type of light spot pixel coordinates or the light spot pixel coordinates at the center position in the same column of pixels.
[0032] Combining steps A to D, this embodiment uses pixel coordinates and distance information to fit modeling parameters, reduces camera detection errors, and overcomes the problem that the image information and distance information collected by the structured light module in the calibration device are not linearly related; and after fitting the modeling parameters, a distance measurement function is further constructed to achieve the effect of simplifying the camera calibration steps, speeding up the acquisition of distance information, and improving the efficiency of the robot using the aforementioned structured light module for distance measurement.
[0033] As an embodiment, after the light strip is imaged by the camera, it is represented by corresponding multiple columns of pixel points in the pixel plane of the camera; each column of pixel points is used to identify the pixel position formed by the light spot of the light strip at each measuring angle in the photosensitive element of the camera; each column of pixel points forms a corresponding measuring angle relative to the optical axis of the camera, which is used to represent the azimuth characteristics of each light spot projected by the light strip on the surface of the object to be measured relative to the optical center of the camera; in this embodiment, the camera can detect in real time the pixel coordinates of the light spots in each column formed by the projection of the light strip in the pixel plane, and each light strip contained in the same frame image captured by the camera can be represented by multiple columns of pixel points in the pixel coordinate system, and the structured light emitted by the structured light emitter can be regarded as composed of multiple single-point line lasers, and the structured light emitter actually emits Figure 3 Although a strip of light is projected onto the surface of the object being measured, the number of columns of pixels formed by the strip of light and the distribution of the columns of pixels projected by the same strip of light on the pixel plane can be determined according to the camera resolution of the structured light module, the operating computing power of the controller of the structured light module, and the specifications of the internal and external parameters of the camera, and this embodiment does not impose any restrictions.
[0034] In this embodiment, each column of pixels forms a corresponding measurement angle relative to the optical axis of the camera. There is a light point pixel coordinate in the pixel coordinates of a column of pixels projected by the light bar. One light point pixel coordinate corresponds to one measurement angle. One measurement angle represents the azimuth formed by a position point on the measured surface relative to the optical axis of the camera.
[0035] It can be understood that the structured light emitted by the structured light emitter changes the optical path through the camera so that the structured light forms a structured light surface, which can be equivalent to being decomposed into multiple single-point line lasers; the light source of the structured light emitter is only one point, and the optical path is changed by the lens and the wave mirror, so that the emitted structured light forms an effect of an approximate plane, which is equivalent to a structured light surface, such as Figure 3 The structured light surface shown; for example, when the structured light is a line laser, the line laser can be regarded as a collection of multiple single-point line lasers divided according to the measurement angle, rather than being composed of multiple single-point line lasers in a physical sense, and is understood as a line laser. In the structured light surface, a single-point line laser forms a corresponding measurement angle relative to the optical axis of the camera, and a single-point line laser forms a corresponding light point pixel coordinate in a calibration column of the pixel plane of the camera; the light strip is the intersection of the structured light surface and the measured surface of the calibration plate. In order to form the light strip, the calibration plate here can also be the measured object, and the measured surface of the calibration plate is the measured surface of the measured object, such as the target surface.
[0036] The structured light emitted by the structured light emitter is regarded as being composed of multiple single-point line lasers, and is also understood to form a structured light surface through the lens of the camera; within the structured light surface, a single-point line laser forms a corresponding measurement angle relative to the optical axis of the camera, that is, the measurement angle formed by a single-point line laser relative to the optical axis of the camera is set in the structured light surface, and is regarded as a laser line divided according to the corresponding measurement angle, and multiple single-point line lasers with continuous angles are divided; a single-point line laser is correspondingly projected in a column of the pixel plane (pixel coordinate system) of the camera to form light point pixel coordinates, so that each column of pixels is located at a pixel position formed by the projection of a single-point line laser in the pixel plane after being reflected by the calibration plate. Since the structured light surface can be approximately regarded as a combination of countless single-point line lasers emitted from the structured light emitter, the fitting calculation can be performed on certain discrete pixel points in the pixel coordinate system according to the step A to construct a structured light ranging relationship or ranging function, so as to fit the corresponding modeling parameters at a finite number of measurement angles, and then obtain the distance information of the measured object at a finite number of measurement angles of the camera based on the ranging function, which can meet some simple contour point ranging application scenarios.
[0037] Preferably, since the structured light emitted by the structured light emitter can be regarded as forming a structured light surface after passing through the lens of the camera, the light strip is located at the intersection of the structured light surface and the measured surface of the calibration plate. Therefore, the projection position of each laser point constituting the light strip in the photosensitive element of the camera can be connected into a horizontal projection line segment in the pixel coordinate system, thereby reflecting that the structured light is a linear structured light and the light strip is parallel to a coordinate axis direction of the world coordinate system, reducing the complexity of the posture conversion required for calibration and improving the efficiency of camera calibration.
[0038] Since each column of pixels identifies the pixel position of each light spot that composes the light strip projected in the camera's photosensitive element, a column of pixels forms a corresponding measurement angle relative to the camera's optical axis, a single-point line laser forms a corresponding measurement angle relative to the camera's optical axis, and a single-point line laser is correspondingly projected in a column of the camera's pixel plane to form a light spot pixel coordinate, so a column of pixels corresponds to a column of light spot pixel coordinates or a light spot pixel coordinate, and is located at a corresponding measurement angle relative to the camera's optical axis. In step A, each column of pixels fits the modeling parameters on the corresponding column and constructs the structured light ranging relationship on the corresponding column based on the fitted modeling parameters. The structured light ranging relationship constructed for each column of pixels in step A corresponds to a measurement angle, and then the ranging function constructed for each column of pixels in step A corresponds to a measurement angle.
[0039] In step A, the test distance corresponding to each measuring angle can be configured as the distance between the line connecting the center of the structured light emitter and the optical center of the camera and the measured surface of the measured object at each measuring angle. It can be understood that when using a camera that satisfies the pinhole imaging principle, each column of pixels on the pixel plane can be approximately regarded as a picture seen at the same vertical angle of the structured light surface. At this vertical angle, there is a single-point line laser "extracted" from the straight-line structured light (the light strip). The vertical angle is the aforementioned measurement angle, and thus, step A can be executed to fit the modeling parameters with the light point pixel coordinates and the test distance in the same column, and then the ranging model on the same column is modeled (corresponding to constructing the structured light ranging relationship or ranging function, and a structured light ranging relationship or ranging function is constructed at a vertical angle). By analogy, if there are N columns of pixels in the pixel plane where the image captured by the camera is located, N light point pixel coordinates may be extracted therefrom; step A is executed in turn for modeling the structured light module composed of the single-point line laser emitter and the camera at each measurement angle, that is, the corresponding modeling parameters are fitted at each measurement angle to avoid calibrating the camera internal parameters first.
[0040] On the basis of the above embodiment, whenever step C calculates the light bar center pixel coordinates at a measurement angle, step D calculates the distance between the measured object and the structured light module at the corresponding measurement angle, representing the distance information at the position of the corresponding measurement angle; wherein, there is at least one light bar center pixel coordinate in a column of light spot pixel coordinates, so that when step D uses the distance measurement function for conversion, the light spot pixel coordinates are configured to be represented by the light bar center pixel coordinates. If there are N columns of light spot pixel coordinates in the pixel plane where the image captured by the camera is located; the single point line laser and the camera at each measurement angle are modeled in turn, and N distance measurement functions are constructed, correspondingly converting N groups of distance information between the measured object and the structured light module, thereby achieving the distance between the position points in each direction covered by the light bar in the measured object and the structured light module through step D.
[0041] As an embodiment, in step C, the light stripe center extraction algorithm is used to extract the center line of the light stripe in the image captured by the structured light module, and multiple light stripe center pixel coordinates are obtained to form a center line point set; in the image including the light stripe, a column of pixel points corresponds to at least one light stripe center pixel coordinate; wherein, the light stripe center extraction algorithm includes a grayscale centroid extraction algorithm, and of course also includes many mature solutions such as the Steger algorithm based on the Hessian matrix, which can relatively accurately extract the light stripe center pixel coordinates while ensuring the accuracy of the added internal parameters, and use the ranging function to speed up the ranging operation.
[0042] In some embodiments, the structured light module can be installed on the body of the robot. After the structured light module is installed, the angle between the center line of the structured light emitted by the structured light emitter (a single-point line laser located at the angular bisector of the emission viewing angle) and the baseline is equivalent to the complementary angle of the angle between the structured light surface and the working surface of the robot; the baseline refers to the line between the center of the structured light emitter and the optical center of the camera when the structured light emitter and the camera are located at the same installation height, and the installation height is the baseline distance. The emission angle of the structured light emitter is related to the detection distance that the robot where the structured light module is located needs to meet, the body width of the robot, and the mechanical distance between the structured light emitter and the camera.
[0043] Of course, if a specific emission angle is required, it can be achieved by adjusting the detection distance that the robot where the structured light module is located needs to meet and the mechanical distance between the structured light emitter and the camera. In some application scenarios, when the detection distance that the robot needs to meet and the robot's body width are determined, the emission angle of the structured light emitter can be changed within a certain angle range by adjusting the mechanical distance between the structured light emitter and the camera.
[0044] As an example, Figure 4As shown, in step A, the method of fitting modeling parameters using pixel coordinates and distance information includes the following steps: Step A1: Control the structured light emitter to emit structured light, project it onto the calibration plate and form a light strip, for example, Figure 3 The structured light emitted by the structured light emitter in the camera changes the optical path through the lens and wave mirror of the camera, so that the structured light emitted by the structured light emitter forms a structured light surface, which can be understood as the effect of the emitted light being an approximate plane; the structured light surface intersects with the surface of the object to be measured to form the light strip shown in the figure, and controls the camera to collect the calibration plate image, including the original image of the light strip, wherein a single-point line laser can be regarded as emitted by a single-point line laser emitter. For example, a single-point line laser is used to emit a single-point line laser, and multiple single-point line lasers emit multiple single-point line lasers. Then a straight line laser can be regarded as emitting to form a laser plane, and its emission The laser plane is regarded as composed of multiple single-point line lasers, and the straight line laser can be equivalently decomposed into multiple single-point line lasers; similarly, the structured light emitter disclosed in the present application can be regarded as composed of multiple single-point line laser emitters, each single-point line laser emitter emits a single-point line laser, and multiple single-point line lasers form a structured light surface, that is, the structured light emitter emits a structured light surface and intersects with the calibration plate to form a light strip. When using single-point laser triangulation ranging, that is, using the pinhole imaging principle (pinhole model) to establish a mathematical model or range, it is necessary to decompose the structured light into multiple single-point line lasers or directly regard it as a single-point line laser, corresponding to the formation of Figure 2 The optical path of the single point line laser is shown.
[0045] In step A1, the calibration plate is also controlled to move relative to the structured light module, and the calibration plate can be driven by a motor to move relative to the camera, so that the camera obtains images of the calibration plate at different distances, and each frame of the calibration plate image includes a light bar. Generally, the calibration plate moves away from the camera or close to the camera in one direction.
[0046] The camera collects calibration plate images at different test distances to obtain multiple frames of calibration plate images. Each frame of the calibration plate image is, in turn, an image of the calibration plate at different distances relative to the structured light module at the same position, and all include light strips, that is, each frame of the calibration plate image includes imaging information of the light strips. And for each frame of calibration plate image collected, the number of pixel columns calibrated in each frame of calibration plate image in the pixel plane of the camera is preferably equal; each time the camera collects one or more frames of calibration plate image, the center coordinates of the light strips of each column are extracted from the same frame of calibration plate image, and the test distance corresponding to the same column of the same frame of calibration plate image is determined, and the test distance can be understood as the distance between the light strip and the structured light module at the corresponding angle or direction; in step A1, the structured light module can actively or under the command of an external controller extract height calibration data, X-axis component or Y-axis component of the light strip point cloud, X-direction calibration data of the black and white area or white area in the calibration plate, etc. from the calibration plate image, which can be used to convert the row pixel position information of the light strip (or the center line of the light strip), or extract the center of gravity position of each column of the light strip, including the center coordinates of the light strip, to provide an input data set for subsequent fitting of modeling parameters.
[0047] In some embodiments, the point cloud that truly expresses the spatial structure of the structured light surface can effectively express the real point cloud spatial coordinates through fitting, thereby ensuring the accuracy of the obtained data.
[0048] Step A2, in the multiple frames of calibration plate images obtained in step A1, nonlinear fitting is performed on the pixel coordinates of the light spots in the same column and the test distances corresponding to the same calibration column to obtain the modeling parameters corresponding to the same column and reduce the fitting error; wherein the same column is a column with the same order from each frame of calibration plate images, which represents the projection position information of the light bar in the pixel position, specifically, the pixel coordinates of the light bar projected at an angle or direction in each frame of calibration plate images, and then nonlinear fitting is performed on the test distances corresponding to the same column of the multiple frames of calibration plate images to obtain the coefficients required for constructing the structured light ranging relationship under the pinhole imaging principle, thereby minimizing the fitting error generated on the same column in each frame of calibration plate images and ensuring the fitting accuracy.
[0049] In this embodiment, the modeling parameters corresponding to the same column are the coefficients required to construct the structured light ranging relationship under the same column. The structured light ranging relationship is a relationship between the pixel coordinates of the light spots in the same column and the corresponding test distances constructed based on the pinhole imaging principle. Constructing / defining the structured light ranging relationship under the pinhole imaging principle can be specifically understood as a relationship between the center coordinates of the light strip constructed under the pinhole imaging principle and the corresponding test distances. The coefficients required to construct the structured light ranging relationship under the same column can be fitted in combination with the data of the calibration plate image at different distances. In this embodiment, the structured light ranging relationship can be used as the corresponding function to be fitted in the nonlinear fitting to fit the modeling parameters, including at least two fitting coefficients.
[0050] In this embodiment, in the process of performing nonlinear fitting on the light spot pixel coordinates in the same column and the test distance corresponding to the same column, the optimal relationship between the light spot pixel coordinates and the test distance is fitted and the modeling parameters are obtained, wherein the light spot pixel coordinates and the test distance (the distance of the calibration plate relative to the structured light module) extracted from the calibration plate image at different distances are known quantities that can be obtained in real time from each frame of the calibration plate image. Therefore, these known quantities can be used to fit the coefficients required to construct the structured light ranging relationship under the same column from the image data of multiple frames of the calibration plate image, thereby establishing a mathematical model for the structured light module ranging.
[0051] For light bars, Figure 3 The center coordinates of the light strip can be extracted from the two-dimensional light strip image of the intersection of the structured light surface in the image and the surface of the object to be measured, including the X-axis component and the Y-axis component, which correspond to the U-axis component (expressed by the number of pixels) and the V-axis component (expressed by the number of pixels) of the pixel coordinate system, which are derived from the result of the one-line structured light projection. In this embodiment, the center coordinates of the light strip can represent the pixel coordinates that are not affected by distortion.
[0052] Therefore, step A2 can calculate the aforementioned modeling parameters by performing nonlinear fitting on the mapping relationship between the light spot pixel coordinates and the test distance in the same column of multiple frames of calibration plate images, as the optimal parameters required to construct the mapping relationship between the light spot pixel coordinates and the test distance in the same column, that is, the coefficients required to construct the structured light ranging relationship. The fitted modeling parameters can replace the traditional camera intrinsic parameters in a direction or a measurement angle relative to the optical axis of the camera, so as to replace or omit the calibration of the camera intrinsic parameters before the ranging operation.
[0053] Step A3, by repeatedly executing step A2 to traverse each column of light spot pixel coordinates and their corresponding test distances in each frame of the calibration plate image obtained in step A1, obtain the modeling parameters corresponding to all columns in turn, and determine that the calibration of the modeling parameters of the structured light module is completed; the modeling parameters corresponding to the same column are the coefficients required to construct the structured light ranging relationship under the same column. The structured light ranging relationship is a relationship between the light spot pixel coordinates in the same column and the corresponding test distances constructed based on the pinhole imaging principle, so that the structured light ranging relationship under the same column forms the function to be fitted corresponding to the nonlinear fitting of step A2.
[0054] Preferably, the number of executions of step A2 is equal to the number of pixel columns calibrated in one frame of the calibration plate image, and the number of calibrated pixel columns in each frame of the calibration plate image is equal.
[0055] The calibrated column pixels in each frame of the calibration plate image are represented by the corresponding multiple columns of pixel points in the pixel plane of the camera after the light strip is imaged by the camera; the pixel positions formed by the light spots of the light strip at each measurement angle in the photosensitive element of the camera, wherein each measurement angle corresponds to one column. In this way, step A3 can traverse all frame calibration plate images in sequence by repeatedly executing step A2 to obtain each column in the same frame calibration plate image and determine the corresponding test distance on the corresponding column, and can also complete the traversal of each column (or each pixel column) in the same frame calibration plate image to extract the pixel coordinates of the light points in each column. In order to obtain the modeling parameters corresponding to the same column, it is necessary to first traverse the pixel coordinates of the light points in the current column of all frame calibration plate images, and then traverse the pixel coordinates of the light points in the next column of all frame calibration plate images. Based on this traversal order, the modeling parameters corresponding to all columns are obtained in sequence through the nonlinear fitting of step A2, and the calibration of the modeling parameters of the structured light module is determined to be completed, including the calibration of the modeling parameters at all measurement angles or all columns, and then the calibration of the structured light module is determined to be completed.
[0056] When the Zhang calibration method disclosed in the prior art is used for camera calibration, a calibration plate with a chessboard is used for camera calibration. The detection error of the chessboard corner points is high, resulting in a large error in the internal parameter matrix solved later. The rotation matrix in the external parameter no longer satisfies the constraints of the special orthogonal group, and the internal and external parameter matrix solved is inaccurate. Combining the above steps 1 to 3, it can be seen that this embodiment relies on controlling the displacement of the calibration plate relative to the structured light module, and uses the center coordinates of the light bars of different calibration columns in the calibration plate image collected at different distances and their corresponding test distances for nonlinear fitting to complete parameter calibration. The fitted modeling parameters can represent the traditional camera internal parameters in a direction or a measurement angle relative to the optical axis of the camera; the modeling parameters are necessary coefficients for constructing the structured light ranging relationship. The structured light ranging relationship can be a deformation of the camera internal parameters based on the pinhole imaging principle (pinhole model, generally used in the single-point line laser reflection light path), which can replace the traditional camera internal parameters to replace or omit the calibration of the camera internal parameters before the ranging operation. Therefore, there is no need to perform additional correction on the camera’s internal parameters, which reduces the amount of calculation and improves the efficiency of structured light module calibration.
[0057] In actual application, after obtaining the modeling parameters, modeling and ranging operations can be performed without calibrating the camera's internal parameters, and the calculation efficiency is improved. In addition, compared with the prior art, step A disclosed in this application does not require a high calibration structure to be built, and the camera calibration model used is simplified, which reduces the ranging cost. It can be widely used in many scenarios such as rapid measurement of product dimensions and rapid guidance of robot trajectories in various fields such as automobile production and electronic manufacturing, and helps the implementation of robot navigation and positioning.
[0058] As an embodiment, before executing the step A1, it includes: adjusting the emission direction of the structured light emitter to form an angle with the optical axis direction of the camera, setting the line between the center of the structured light emitter and the center of the camera to be parallel to the measured surface of the calibration plate, which can be fixed by a special fixture. Note that it is not limited that the camera and the calibration plate must be parallel, nor is it limited that the intersection line of the structured light surface emitted by the structured light emitter and the calibration plate must be horizontal. Among them, the angle between the emission direction of the structured light emitter and the optical axis direction of the camera is also fixed by the fixture, which is used to project structured light and collect images on the same calibration plate, regardless of whether the calibration plate is displaced; the relative position between the center of the structured light emitter and the center of the camera, and the angle between the emission direction of the structured light emitter and the optical axis direction of the camera are fed back to the controller of the structured light module, and the controller of the structured light module can fix the relative position and angle by controlling the fixture, so as to keep the relative posture between the camera and the structured light emitter unchanged.
[0059] It should be noted that the above description is only for the convenience of explaining the relative positions of the components in the structured light module. When the relative positions remain unchanged, the structured light module and the calibration plate do not necessarily have to be placed on the horizontal plane, but can also be placed on the vertical plane. When step A2 is subsequently executed, the calibration plate is driven by the motor to move freely or rotate relative to the slide rail, and the placement posture is unconstrained.
[0060] Combination Figure 2 and Figure 3 It can be seen that the controller of the structured light module sets the emission direction of the structured light emitter to intersect with the calibration plate so that the structured light emitter Figure 2 The structured light (generally a single light ray, such as a single point line laser equivalently decomposed from the structured light surface, which can be referred to as the optical path of the single point line laser) emitted from the calibration plate is incident on the calibration plate at a certain angle to the normal of the measured surface of the calibration plate, and the reflected light is imaged into the camera's photosensitive element through the camera lens, and the camera collects the image formed by the reflected light, wherein the imaging of the reflected light of the structured light collected by the camera in the camera's photosensitive element can be approximately regarded as a combination of various imaging positions formed in the camera's imaging plane after the multiple single point line lasers constituting the structured light are reflected by the calibration plate; the imaging of the reflected light of the structured light collected by the camera in the camera's photosensitive element is represented by the imaging of the light strip in the camera's imaging plane. The imaging of the light strip in the camera's imaging plane is a part of the calibration plate image.
[0061] When the structured light transmitter uses a line laser transmitter, the emitted laser line is structured light, such as multiple single-point line lasers divided approximately, which change the light path through the camera lens to form a laser plane, corresponding to Figure 3 The structured light surface shown in the figure is formed by the intersection of the laser plane and the calibration plate, so that the light strip is a structured light strip, such as a projection spot of a straight line laser in the calibration plate; it can be extracted from the calibration plate image to perform a triangulation operation (constructing similar triangles for distance calculation), specifically, each pixel point on the light strip (the projection position of each single-point line laser in the pixel plane or the imaging position in the imaging plane) can perform a triangulation operation (using the aforementioned modeling parameters to construct a structured light ranging relationship for triangulation calculation) to obtain the distance information of each pixel position within the angle range covered by the light strip relative to the camera.
[0062] On the basis of the above embodiment, when the surface of the camera's photosensitive element (photosensitive surface) is preferably parallel to the measured surface of the calibration plate, the object plane (from the calibration plate) and the image plane (the camera's photosensitive element) can remain parallel, which is convenient for establishing similar triangles using the pinhole imaging principle for triangular ranging calculation, thereby utilizing the triangulation ranging method commonly used in single-point line laser ranging in combination with a nonlinear fitting method (such as the least squares method) to speed up the calibration of modeling parameters.
[0063] The plane where the calibration plate image captured by the camera is located can be regarded as being decomposed into columns. Theoretically, the parameters of the columns are fitted independently, which is equivalent to decomposing the structured light surface into multiple single-point line lasers (structured light emitted by the single-point line laser emitter). At this time, the structured light surface can be approximately decomposed into multiple single-point line lasers. For example, the structured light (straight line light) is divided into single-point line lasers in multiple directions according to a preset angle, and then the triangulation method (pinhole imaging principle) and nonlinear fitting are used for each single-point line laser to speed up the calibration of the pixel coordinates of the same calibration column.
[0064] When the structured light emitter uses a laser emitter, the emitted laser line is structured light. For example, multiple single-point line lasers are composed of countless laser lines emitted by the same line laser emitter to form a laser plane, corresponding to Figure 3 The structured light surface shown in the figure, the intersection of the laser plane and the calibration plate forms a light strip, so that the light strip is a linear structured light strip, such as a projection spot of a straight line laser in the calibration plate; it can be extracted from the calibration plate image to perform a triangulation operation (constructing similar triangles for distance calculation), specifically, each pixel point on the light strip (the projection position of each single-point line laser in the pixel plane or the imaging position in the imaging plane) can perform a triangulation operation (using the aforementioned modeling parameters to construct a structured light ranging relationship for triangulation calculation) to obtain the distance information of each pixel position within the angle range covered by the light strip relative to the camera. In this way, on the basis of keeping the relative posture between the camera and the structured light emitter unchanged, the distance information of each position of the projected structured light in the calibration plate can be accelerated by using the pinhole imaging principle, especially when the structured light emitter is approximately decomposed into multiple single-point line laser emitters, the ranging information at multiple angles can be obtained.
[0065] On the basis of the above embodiment, the method of controlling the displacement of the calibration plate relative to the structured light module in step A1 includes: the controller of the structured light module can drive the calibration plate to slide on the slide rail through a motor; under the drive of the motor, the calibration plate slides a certain distance along the slide rail, the camera collects the calibration plate image once, and the light bar is offset once on the calibration plate, which is an offset in the height direction of the calibration plate, and the offset distance can be detected by the structured light module in real time; the line between the center of the structured light emitter and the optical center of the camera is set as the baseline; the relative position between the structured light emitter and the camera is fixed; the distance between the baseline and the calibration plate is obtained in advance; the relative position between the structured light emitter and the camera is fixed by a fixture; if the structured light emitter is regarded as being decomposed into multiple single-point line laser emitters and each single-point line laser emitter emits a single-point line laser, the fixture fixes the emission direction of each single-point line laser emitter to point to the corresponding measurement angle, and the line between the center of each single-point line laser emitter and the optical center of the camera is set as the baseline on the corresponding measurement angle. Each time the calibration plate image is collected, the calibration plate is located at a different position on the slide rail. When the structured light emitter is turned on, the camera collects an image containing the light strip, and the camera detects in real time the pixel coordinates of the light spots formed by the projection of the light strip in the pixel plane.
[0066] The structured light module is fixedly mounted at one end of the slide rail, and the initial position of the calibration plate is set at the other end of the slide rail, wherein the calibration plate is slidably set on the slide rail. Each time the calibration plate image is collected, the calibration plate is located at a different position on the slide rail, and when the structured light emitter is turned on, the camera collects an image containing the light strip, and the camera detects the pixel coordinates of the light spots formed by the projection of the light strip in the pixel plane in real time, and extracts the pixel coordinates of the light spots formed by the light strip in the pixel plane from the calibration plate image, and records the test distance at the corresponding measurement angle, which are used together to participate in the nonlinear fitting of step A2.
[0067] Specifically, when the calibration plate is set on the normal plane of the slide rail, the height direction of the calibration plate can be regarded as the vertical direction of the slide rail, or parallel to the normal plane of the slide rail, and can also be used as the longitudinal direction of the imaging plane or pixel plane of the camera; preferably, the calibration plate is slidably mounted on one end of the slide rail, and the structured light module is fixedly mounted on the other end of the slide rail. Figure 2 As shown, the calibration plate is installed on the right side of the photosensitive element and the calibration plate is supported to slide left and right on the slide rail. The calibration plate is installed at the right end of the slide rail, and the structured light module is installed at the left end of the slide rail. Specifically, the structured light emitter and the camera are both installed at the left end of the slide rail, and the relative position between the structured light emitter and the camera is fixed by the fixture. Figure 2The line between the center A of the structured light emitter and the optical center B of the camera is set as the baseline AB and is fixed, at least in the process of calibrating the parameters of the camera and the structured light; the slide rail does not have to be placed on a horizontal plane, but can also be placed on a vertical plane and can be rotated at will. Every time the calibration plate slides a certain distance along the slide rail, the light bar on the calibration plate is translated once along the vertical direction of the slide rail; combined with Figure 2 It can be seen that the calibration plate starts to slide to the right from a starting position of the slide rail (set at a track position at a known distance from the camera). When the calibration plate slides to the position where the structured light emitted by the structured light emitter (from a two-dimensional plane perspective, generally from the perspective of the normal plane of the measured surface of the calibration plate, it is regarded as emitting a single-point line laser) intersects the measured surface of the calibration plate at point O, the reflected light at position point O is imaged to position E on the surface of the camera's photosensitive element through the optical center B of the camera's lens, so that the camera obtains the pixel coordinates at position E in real time, which are expressed as the pixel coordinates of the light spot projected by the light bar passing through position point O; at the same time, the vertical distance CO from point O to the baseline AB is also determined, which is expressed as a test distance, which is also the distance between the calibration plate at the current position and the baseline AB, and can be expressed by the displacement of the calibration plate relative to the structured light module along the slide rail. At this time, the angle between the structured light emitted by the structured light emitter (from the perspective of a two-dimensional plane, generally from the perspective of the normal plane of the measured surface of the calibration plate, it is regarded as emitting a single-point line laser) and the baseline AB is ɑ, and the baseline AB is the line connecting the optical center B of the camera and the center A of the structured light emitter. When the calibration plate is set on the normal plane of the slide rail, the baseline AB is parallel to the measured surface of the calibration plate (regarded as perpendicular to the slide rail); then, when the angle between the structured light emitted by the structured light emitter (from the perspective of a two-dimensional plane, generally from the perspective of the normal plane of the measured surface of the calibration plate, it is regarded as emitting a single-point line laser) and the baseline AB is fixed at ɑ, the calibration plate is only driven to slide away from the camera until the structured light emitted by the structured light emitter (from the perspective of a two-dimensional plane, generally from the perspective of the normal plane of the measured surface of the calibration plate, When the laser beam (considered as emitting a single point line laser) intersects the measured surface of the calibration plate at point O', the distance between point O' and point O is equal to y, and the reflected light at point O' is imaged to position F on the surface of the camera's photosensitive element through the optical center B of the camera lens, that is, in the process of the calibration plate deviating from the fixed position of the camera along the slide rail, the imaging position of the reflected light on the surface of the camera's photosensitive element is offset from position E to position F along the height direction, and the camera obtains the pixel coordinates at position F in real time, which are expressed as the pixel coordinates of the light spot projected by the calibration plate at point O'; at the same time, the vertical distance C'O' from point O' to the baseline AB is also determined, which is expressed as a test distance, and is also the distance between the calibration plate at the current position and the baseline AB, which can be expressed by the displacement of the calibration plate relative to the structured light module along the slide rail.
[0068] On this basis, the distance CO and the pixel coordinates at the position E can be used as a vector group to participate in the nonlinear fitting of step A2, and the distance C'O' and the pixel coordinates at the position F can also be used as a vector group to participate in the nonlinear fitting of step A2, and the modeling parameters under the same column are fitted together. Therefore, the calibration plate calibrates the structured light module under the drive of the motor, and there is no need to specially set up targets such as checkerboards. The calibration plate only needs to slide along the slide rail controlled by the motor to collect the distance information and pixel coordinates required for fitting the corresponding ranging relationship under the pinhole imaging model, which is easy to realize automation; thereby, while ensuring the reliability of the calibration results of the structured light module, the efficiency of the structured light module calibration is improved and the calibration error is reduced.
[0069] It should be noted that the size of the calibration plate and the length of the slide rail should be adaptively designed according to the relative position between the camera and the structured light emitter and the ranging range limited on the slide rail to ensure that the posture information within the effective measurement range of the structured light module is covered.
[0070] Preferably, when the camera is a distortion-free camera and the surface of the camera's photosensitive element forms a certain oblique angle with the measured surface of the calibration plate, the distance between the baseline and the calibration plate can be directly calculated according to the pinhole imaging principle without affecting the calibration accuracy of the camera.
[0071] As an embodiment, in the step A1, during the sliding of the calibration plate along the slide rail, each time the calibration plate slides a preset fixed distance, the camera captures an image of the light bar, and the imaging position of the light bar in the imaging plane is shifted once along the height direction, and each light spot position may be shifted along the height direction by a preset height, that is, the imaging position in the image captured before the current sliding is shifted along the height direction by a preset height, and then, the light spot pixel coordinates at the corresponding measurement angle are shifted in the opposite direction of the height direction in the pixel plane by a preset longitudinal offset, specifically, the pixel coordinates of each light spot formed by the projection of the light bar in the pixel plane are shifted in the opposite direction of the height direction by a preset longitudinal offset, which is expressed in the number of pixels and detected by the camera in real time.
[0072] In some embodiments, when the calibration plate approaches the structured light module, that is, when it approaches the structured light module in steps according to a preset fixed distance, the calibration plate slides along the slide rail from the upper limit of the range to the lower limit of the range with a preset fixed distance as the stepping distance. The imaging position of the light bar in the imaging plane is offset downward along the longitudinal axis (in the height direction) with a preset fixed height as the stepping distance, which can be detected by the camera in real time; the pixel coordinates of the light spot formed by the projection of the light bar in the pixel plane are offset upward along the longitudinal axis (in the opposite direction of the height direction) with a preset longitudinal offset as the stepping distance, and the camera also detects the pixel coordinates of the light spot in the pixel coordinate system or pixel plane, preferably starting from a pixel offset starting point in the pixel plane of the camera and measured with a preset longitudinal offset as the stepping distance.
[0073] In other embodiments, when the calibration plate is away from the structured light module, that is, when it moves away from the structured light module in steps according to a preset fixed distance, the calibration plate slides along the slide rail from the lower limit of the range to the upper limit of the range with a preset fixed distance as the stepping distance, and the imaging position of the light bar in the imaging plane is offset upward along the longitudinal axis (height direction) with a preset fixed height as the stepping distance, which can be detected by the camera in real time; the pixel coordinates of the light spot formed by the projection of the light bar in the pixel plane are offset downward along the longitudinal axis (in the opposite direction of the height direction) with a fixed longitudinal offset as the stepping distance, and the camera also detects the pixel coordinates of the light spot in the pixel coordinate system or pixel plane, preferably starting from a pixel offset starting point in the pixel plane of the camera, and measured with a preset longitudinal offset as the stepping distance.
[0074] The upper limit of the range of the slide rail is the position of the calibration plate on the slide rail that is farthest from the structured light module (also considered as the camera), and the distance relative to the structured light module or the baseline is obtained in advance. The lower limit of the range of the slide rail is the position of the calibration plate on the slide rail that is closest to the structured light module (also considered as the camera), and the distance relative to the structured light module or the baseline is obtained in advance.
[0075] When the calibration plate slides to a position on the slide rail, which may be when the initial position of the calibration plate is placed at the upper limit or the lower limit of the range, the offset formed by the imaging position of the light bar in the imaging plane relative to the reference position is marked as the light bar offset, specifically, the offset formed by the imaging position of the light bar in the imaging plane toward the same measuring angle relative to the reference position at the same measuring angle is marked as the light bar offset at the same measuring angle, that is, the displacement caused by the reference position deviating from the corresponding column is detected in the imaging plane coordinate system along the corresponding column of the measuring angle, wherein the reference position is, within the detectable area of the structured light module, the imaging position of the reflected light on the measured surface of the measured object farthest from the structured light module emitted by the structured light emitter in the camera, recorded as the limit position of the reflected light imaging on the surface of the photosensitive element when the measured surface of the measured object is infinitely far away from the baseline; the measured object includes the calibration plate. The reference position at the corresponding measurement angle is set to be within the detectable area of the structured light module. At the corresponding measurement angle, the imaging position of the reflected light on the measured surface of the measured object farthest from the structured light module in the camera. Preferably, the line connecting the reference position at the corresponding measurement angle and the optical center of the camera is parallel to a single-point line laser at the same measurement angle emitted by the structured light emitter. In summary, at each measurement angle, the reference position is used to represent the offset reference point of the projection position of the light bar in the imaging plane coordinate system, and provides an imaging position offset starting point for the measured object or calibration plate at any distance, so as to facilitate the measurement and description of the displacement of the light bar in the imaging plane or the photosensitive element from the reference position. Moreover, during the sliding process of the calibration plate, by setting the preset longitudinal offset as the stepping distance, it is convenient to measure the pixel coordinates of the detection light point starting from a pixel offset starting point in the pixel plane of the camera.
[0076] Since the calibration plate will slide from the initial position to a position, the distance it slides and the position reached by the calibration plate are obtained in real time to determine the distance between the baseline and the calibration plate; since the imaging position of the light bar in the imaging plane is offset along the height direction with a preset fixed height as a step distance, the imaging position of the light bar in the imaging plane is obtained in real time, specifically, it is determined by calculating the coordinate information of the imaging position of the light bar reflected from the calibration plate at the initial position in the imaging plane; since the pixel coordinates of the light spot formed by the projection of the light bar in the pixel plane are offset in the opposite direction of the height direction with a preset longitudinal offset as a step distance, the pixel coordinates of the light spot formed by the projection of the light bar in the pixel plane are obtained in real time, specifically, it is determined by calculating the coordinate information of the pixel position projected from the light bar reflected from the calibration plate at the initial position in the pixel plane. Therefore, when the relative position of the structured light emitter and the camera remains unchanged, when the calibration plate is displaced relative to the baseline, the light bar offset changes, and the pixel coordinates of the light spot change.
[0077] refer to Figure 2 It can be known that when the calibration plate slides until the structured light emitted by the structured light emitter (from the perspective of a two-dimensional plane, generally from the perspective of the normal plane of the measured surface of the calibration plate, it is regarded as emitting a single-point line laser) intersects the calibration plate at position point O, and when the intersection line of the structured light surface formed by the structured light emitter and the measured surface of the calibration plate passes through position point O, the offset of the imaging position E of the light bar in the imaging plane relative to the reference position D is the light bar offset DE, represented by x in the figure; wherein, during the sliding of the calibration plate on the slide rail, the line between the center of the structured light emitter and the center of the camera remains parallel to the calibration plate. The motor driving the calibration plate to slide is a stepper motor. At each position where the calibration plate slides driven by the stepper motor, the distance of each sliding of the calibration plate is the stepping distance generated by the stepper motor control. The stepping distance is a preset fixed distance. Therefore, when the calibration plate slides along the slide rail for the same distance, the distance of the calibration plate relative to the structured light module and its variation are both measurable and recorded. The offset of the light bar on the calibration plate is equal and can be within the visible range of the camera, which is also equivalent to the detectable area of the structured light module. The light bar offset and its variation are both detected in real time, and its light spot pixel coordinates are obtained, so as to obtain multiple sets of vectors for nonlinear fitting in step A2, each set of vectors is a column of light spot pixel coordinates of a frame of calibration plate image and its corresponding test distance, and multiple sets of vectors are the light spot pixel coordinates of the same column of all frames of calibration plate images and their corresponding test distances, so as to improve the fitting accuracy of the modeling parameters corresponding to the same column.
[0078] In some embodiments, on the slide rail, the initial position of the calibration plate is placed at the lower limit of the range, that is, the structured light module is fixedly mounted at one end of the slide rail, and the calibration plate is slidably mounted on the other end of the slide rail, and is driven by a stepper motor to step toward the upper limit of the range of the slide rail in sequence. Here, the constraint on the range measurement range defined on the slide rail can be determined based on whether the light bar is clearly imaged on the photosensitive element of the camera, at least to ensure that the image coordinate position (the coordinate position in the image coordinate system (i.e., the imaging plane coordinate system)) projected by the light bar does not deviate from the surface area of the photosensitive element, such as the farthest position from the camera that the calibration plate is allowed to slide on the slide rail is the farthest position and the closest position that supports the light bar projected on the calibration plate by the structured light emitted by the structured light emitter for focusing imaging in the camera. In some embodiments, when the calibration plate is at the upper limit of the range of the slide rail, the imaging position of the light bar in the imaging plane is set as the reference position, wherein the upper limit of the range of the slide rail is the farthest position from the structured light module (which can also be regarded as the camera) that the calibration plate is allowed to slide on the slide rail.
[0079] As an embodiment, in the same column, the method for constructing the structured light ranging relationship includes, which can also be understood as the method for constructing the structured light ranging relationship at the same measurement angle includes, First, the mathematical model between the light point pixel coordinates and the test distance to be established (i.e. the aforementioned structured light ranging relationship) is explained. For ease of understanding, the single-point laser triangulation ranging principle is explained first. From the perspective of a two-dimensional plane, generally the perspective of the normal plane of the measured surface of the calibration plate, the structured light emitted by the structured light emitter is Figure 2 In the figure, a single-point line laser is emitted, which is equivalent to the single-point line laser optical path diagram. When the structured light emitted by the structured light emitter is incident on the calibration plate at a certain angle to the normal of the measured surface of the calibration plate, the reflected light is imaged into the photosensitive element of the camera through the lens at position B. B is the optical center of the camera and also the pinhole in the pinhole model.
[0080] like Figure 2 As shown, in this embodiment, the line between the center A of the structured light emitter and the optical center B of the camera is set as the baseline, and the length of the baseline is set as the baseline distance AB; the baseline distance AB is predetermined, is the assembly parameter of the structured light module, and is also the assembly parameter in the longitudinal direction. If the structured light emitter is decomposed into multiple single-point line laser emitters and each single-point line laser emitter emits a single-point line laser, the fixture fixes the emission direction of each single-point line laser emitter to point to the corresponding measurement angle, and the line between the center of each single-point line laser emitter and the optical center of the camera is set as the baseline at the corresponding measurement angle.
[0081] In this embodiment, the distance between the baseline and the calibration plate is set as the test distance, specifically, the distance between the baseline and the calibration plate at the corresponding measurement angle is set as the test distance d at the corresponding measurement angle. When the calibration plate slides to the position where the structured light emitted by the structured light emitter (from the perspective of a two-dimensional plane, generally from the perspective of the normal plane of the measured surface of the calibration plate, it is regarded as emitting a single point line laser) intersects the measured surface of the calibration plate at point O, the reflected light at position point O is imaged to position E on the surface of the camera's photosensitive element through the optical center B of the camera's lens, so that the camera obtains the pixel coordinates at position E in real time, which are expressed as the pixel coordinates of the light point projected by the light bar passing through position point O. Figure 2 The distance OC represents the test distance at the corresponding measurement angle, which is used as a directly measurable quantity in step A1. In some embodiments, when the distance between the structured light module and the calibration plate is represented by the horizontal distance between the line between the center of the structured light emitter and the optical center of the camera and the measured surface of the calibration plate, it can represent the displacement of the calibration plate relative to the structured light module along the slide rail when the slide rail is placed horizontally, and can also represent the horizontal component of the displacement of the calibration plate relative to the structured light module along the slide rail when the slide rail is placed tilted to the horizontal plane.
[0082] The structured light ranging relationship constructed according to the pinhole imaging principle is: Px=k0 / d+k1, which forms a relationship between the pixel coordinates of the light spot and the test distance in the orientation represented by a column of pixels, which is actually converted into a relationship between the pixel coordinates of the light spot and the test distance. Among them, Px is the pixel coordinates of the light spot at the corresponding measurement angle, which is analyzed here by the X-axis component (associated with the baseline distance AB), the unit is pixel, and it is expressed as the number of pixels. The modeling parameters corresponding to the same column include k0 and k1, that is, the coefficients required to construct the structured light ranging relationship at the corresponding measurement angle are k0 and k1 respectively. Specifically, during the sliding of the calibration plate on the slide rail, the pixel coordinates Px of the light spot, the baseline distance AB, the focal length f of the camera and the test distance OC are all obtained in advance or calculated in advance, and the emission angle of the structured light emitter and the optical axis direction of the camera are predetermined to determine the reflection light path of the structured light.
[0083] At the same measurement angle, the ratio of the baseline distance to the test distance is equal to the ratio of the light bar offset to the focal length of the camera. Figure 2 It can be seen that the position D on the surface of the camera's photosensitive element is used as the reference position, where the line connecting the reference position D and the camera's optical center B is parallel to the structured light emitted by the structured light emitter at position A (from a two-dimensional plane perspective, generally from the perspective of the normal plane of the measured surface of the calibration plate, it is regarded as emitting a single point line laser), and the angle ɑ formed by the structured light emitted by the structured light emitter and the baseline AB (also regarded as the baseline distance AB) also determines the reference position. Figure 2 It can be seen that when the calibration plate slides until the structured light emitted by the structured light emitter (from the perspective of the two-dimensional plane, generally from the perspective of the normal plane of the measured surface of the calibration plate, it is regarded as emitting a single point line laser) intersects with the calibration plate at position point O, and when the intersection line of the structured light surface formed by the structured light emitter and the measured surface of the calibration plate passes through position point O, the offset formed by the imaging position E of the light bar in the imaging plane relative to the reference position D is the light bar offset DE, which is represented by x in the figure; the distance between the baseline AB and the calibration plate, that is, the test distance is represented by OC. Since the baseline AB is parallel to the calibration plate, the surface of the camera's photosensitive element is regarded as parallel to the baseline, so based on the pinhole imaging principle, the triangle ABO and the triangle DEB are similar, so there is: AB / OC=x / f, which simplifies the pinhole imaging model. Therefore, x=(AB*f) / OC, where OC supports real-time measurement during the sliding of the calibration plate on the slide rail. Thus, a calculation model for the light bar offset is established through the baseline distance, focal length and test distance.
[0084] At the same measurement angle, the sum of the product of the light spot pixel coordinates Px and the single pixel size CellSize and the projection error Dev is equal to the light strip offset x, that is, x=Px*CellSize+Dev. The premise for this relationship is that the imaging of the light strip in the imaging plane of the camera is set in the imaging plane coordinate system, wherein the reference position and the light strip offset are both set in the imaging plane coordinate system, the light spot pixel coordinates are set in the pixel coordinate system, and the pixel coordinate system is the coordinate system of the pixel plane of the camera. The projection error Dev represents a light strip on the surface of the object being measured. The difference between the light strip offset acquired by the camera in real time and the light strip offset formed by the actual projection in the imaging plane of the camera is the result obtained in advance experiments. Figure 2 In the equation, combined with x=(AB*f) / OC, the structured light ranging relationship that can be obtained is Px=[(AB*f / OC)-Dev] / CellSize; then, k0 is represented by (AB*f) / CellSize; at the same time, k0 is represented by -Dev / CellSize; at this point, the structured light ranging relationship can be simplified to Px=k0 / d+k1. In this embodiment, k0 and k1 are both modeling parameters to be determined. Therefore, the camera intrinsic parameters that originally need to be calibrated are simplified from Dev, CellSize, AB and f to k0 and k1, which not only simplifies the ranging relationship that requires coordinate system conversion under the pinhole imaging principle, but also reduces the parameters that need to be calibrated. The simplest functional relationship between the pixel coordinates of the light spot in the photosensitive element and the test distance (the distance between the measured surface (target surface) of the calibration plate and the baseline) is formed.
[0085] In summary, the size of a single pixel and the projection error are both unknown quantities, so k0 and k1 constitute unknown quantities in the structured light ranging equation, which need to be solved by sliding the calibration plate. Figure 2 The triangular ranging relationship constructed by triangle ABO and triangle DEB is simplified, reducing the type and number of parameters that need to be calculated, which is equivalent to reducing the number of camera intrinsic parameters that need to be calibrated, simplifying the camera calibration method. Therefore, it is only necessary to determine the two parameters k0 and k1 in the aforementioned structured light ranging relationship to complete the modeling of the structured light module, that is, to establish a model that uses structured light to measure the distance between the baseline and the object being measured, which is equivalent to completing the calibration of the camera, and has simplified the camera calibration method. Then, on the basis of simplifying the camera intrinsic calibration operation, the functional relationship between the test distance and the light point pixel coordinates d=k0 / (Px-k1) is determined, which is used to calculate the test distance when the light point pixel coordinates are known, as the ranging information of the structured light module.
[0086] As an embodiment, in the step A, the method for constructing a ranging function with modeling parameters as known quantities includes: when it is determined that the structured light ranging relationship at the corresponding measurement angle is Px=k0 / d+k1, the modeling parameters required for the structured light ranging relationship at the corresponding measurement angle are all set to known quantities, and the light point pixel coordinates Px at the corresponding measurement angle and the test distance d at the corresponding measurement angle are both set to unknown quantities; then Px=k0 / d+k1 is converted to d=k0 / (Px-k1), and the converted d=k0 / (Px-k1) is recorded as the ranging function at the corresponding measurement angle, which is used to represent the test distance d at the corresponding measurement angle and The relationship between the light spot pixel coordinates Px is determined, and the construction of the ranging function at the corresponding measurement angle is completed, wherein k0 and k1 are both changed into known quantities, so that when the light strip center pixel coordinates calculated in step C are configured as the light spot pixel coordinates Px, the test distance d is converted by the ranging function, and the test distance is represented in step D as the distance between the object under test and the structured light module at the corresponding measurement angle; the specific implementation is that whenever step C calculates the light strip center pixel coordinates at a measurement angle, step D converts the distance between the object under test and the structured light module at the corresponding measurement angle, which represents the distance information at the orientation of the corresponding measurement angle.
[0087] The premise for constructing the structured light ranging relationship in this application is that it is necessary to understand how the camera itself obtains the world coordinates in three-dimensional space step by step from the pixel coordinates. The following coordinate systems are involved here: World coordinate system: A reference coordinate system is also selected in the environment to describe the position of the camera and the object being measured. This coordinate system is called the world coordinate system. The relationship between the camera coordinate system and the world coordinate system can be described by a rotation matrix and a translation vector. In this application, the coordinate system where the structured light module or structured light emitter is located can be used as the world coordinate system.
[0088] Camera coordinate system: The origin of the camera coordinate system is the optical center of the camera. The x-axis and y-axis are parallel to the X and Y axes of the image. The z-axis is the optical axis of the camera, which is perpendicular to the image plane. The spatial rectangular coordinate system formed by this is called the camera coordinate system, also known as the camera coordinate system. The camera coordinate system is a three-dimensional coordinate system. The intersection of the optical axis and the image plane is the origin of the image coordinate system. The rectangular coordinate system formed by the X and Y axes of the image is the image coordinate system. The image coordinate system is a two-dimensional coordinate system. If the world coordinates are converted to camera coordinates, there is a rigid body transformation. In fact, it corresponds to the camera extrinsic parameters. The camera coordinates are converted to image coordinates, which is a 3D->2D perspective projection process.
[0089] Imaging plane coordinate system (X, Y): The pixel coordinate system is not conducive to coordinate transformation, so it is necessary to establish an image coordinate system XOY (i.e., imaging plane coordinate system). The unit of its coordinate axis is usually millimeter (mm). The origin is the intersection of the camera optical axis and the image plane (called the principal point), i.e., the center point of the image. The X axis and Y axis are parallel to the U axis and V axis respectively. Therefore, the two coordinate systems are actually in a translation relationship, that is, they can be obtained by translation.
[0090] The pixel coordinate system (U, V) is a two-dimensional rectangular coordinate system that reflects the arrangement of pixels on the two-dimensional image plane. The U axis and the V axis are parallel to the two sides of the image plane. The unit of the coordinate axis in the pixel coordinate system is pixel (integer). The image coordinates are converted into pixel coordinates, and the image coordinate system is discretized. The origin of the pixel coordinate system is generally located in the upper left corner of the image. The offset from the optical center to the origin of the pixel coordinate system is recorded as the reference pixel coordinate, that is, the coordinate of the camera optical center in the pixel coordinate system. The size of each pixel in the horizontal and vertical directions is recorded as (mm / pixel). The picture is composed of pixels, and each pixel can be represented by an ordered binary group (u, v). The pixel coordinate system is a two-dimensional coordinate system that identifies the position of each pixel in the photosensitive element.
[0091] As an embodiment, under the control of the controller of the aforementioned structured light module, the motor drives the calibration plate to slide on the slide rail. During the sliding process of the calibration plate along the slide rail, the calibration plate slides through multiple positions in sequence at least according to a preset fixed distance (a stepping distance driven by the motor), all of which are located between the upper limit of the range of the slide rail and the lower limit of the range of the slide rail; at each position, the camera captures a frame of the calibration plate image and extracts the pixel coordinates of the light spots in the corresponding column, including the center coordinates of the light strips in a corresponding column in the pixel coordinate system; when the relative position of the structured light emitter and the camera remains unchanged, when the calibration plate is displaced relative to the baseline, the light strip offset changes, and the pixel coordinates of the light spots change. Therefore, in step A2, the method for nonlinear fitting of the pixel coordinates of the light spots in the same column and their corresponding test distances includes: Step 21, sequentially traverse the i-th column of the M frames of calibration plate images collected in step A1; then execute step 22. The i-th column is a column of pixel positions projected by the light spot of the light strip at the corresponding measurement angle in the photosensitive element of the camera, and is recorded as the calibration column; generally, it starts from the first column of pixel points of the first frame of the calibration plate image collected by the camera at the initial position of the slide rail.
[0092] Step 22, extract the row pixel position coordinate Pxj of the center of the light strip of the i-th column from the current frame of calibration plate image, and record it as the pixel coordinate of the light spot of the i-th column of the current frame of calibration plate image. The row pixel position coordinate of the center of the light strip of the i-th column can be understood as the coordinate position of the center of gravity of the light strip of the i-th column in the pixel plane, which can be extracted in real time or obtained by calculation through relevant algorithms; at the same time, determine the test distance dj corresponding to the i-th column of the same frame of calibration plate image, which can be measured from the slide rail; j represents the sorting number of the current frame of calibration plate image, so that the current frame of calibration plate image is marked as the j-th frame of calibration plate image; j is a positive integer, j is greater than 0, j is less than or equal to M, M is also a positive integer, preferably 8 or greater. Then execute step 23.
[0093] Step 23, configure j=j+1, so that the data of the j+1th frame calibration plate image is updated to the data of the jth frame calibration plate image, and then repeat step 22 until the i-th column of the M frames of calibration plate images is traversed, and the i-th column light point pixel coordinates in each frame calibration plate image are combined into a coordinate set {Px1, Px2,…, PxM}, and at the same time, the test distances corresponding to the i-th column in each frame calibration plate image are combined into a distance set {d1, d2,…, dM} or a distance set {1 / d1, 1 / d2,…, 1 / dM} to adapt to the aforementioned relationship Px=k0 / d+k1, and then execute step 24.
[0094] In step 23, k0 and k1 are both unknown quantities, Px and d are both known quantities, Px includes Px1, Px2, ..., PxM; d includes d1, d2, ..., dM. Among them, j in step 21 is incremented from the value 1, so the configuration of step 23 j=j+1 is equivalent to traversing the next frame of the calibration plate image in step 21.
[0095] Preferably, the center coordinates of any two light bars in {Px1, Px2, ..., PxM} are not equal, and any two test distances in {d1, d2, ..., dM} are not equal.
[0096] Step 24, configure the coordinate set of step 23 and the distance set of step 23 as data sets on two coordinate axes (i.e., two sets of input data sets of the function to be fitted), perform nonlinear fitting, obtain the corresponding modeling parameters k0 and k1 of the i-th column, determine that the calibration of the modeling parameters corresponding to the i-th column is completed; and determine to fit the structured light ranging relationship corresponding to the i-th column. Whenever the structured light ranging relationship corresponding to the i-th column is fitted, it is determined to find the relationship between the light bar center coordinate Px and the test distance d, and it is not necessary to calibrate the camera internal parameters in advance, or it is understood to simplify the camera calibration process; and record the modeling parameters k0 and k1 corresponding to the i-th column as a modeling parameter pair of the i-th column, i.e., the modeling parameters corresponding to the i-th column. At this time, in the aforementioned relationship Px=k0 / d+k1, k0 and k1 become known quantities, while Px and d become unknown quantities. Then execute step A3.
[0097] The modeling parameters k0 and k1 corresponding to the same column fitted in step 24 satisfy the structured light ranging relationship corresponding to the same column collected by the calibration plate moving along the slide rail, and are further extended to meet the ranging requirements of the structured light module for the object being measured at the corresponding measurement angle, that is, the fitting coefficients required for the ranging model of the structured light module at the corresponding measurement angle.
[0098] It should be noted that the modeling parameters fitted by the modeling parameter calibration method are necessary parameters in the structured light ranging relationship followed by the same column of each frame of the calibration plate image and the corresponding light strip center coordinates during the sliding process of the calibration plate. In order to construct this structured light ranging relationship on the same column of each frame of the calibration plate image, it is necessary to introduce the light strip center coordinates projected at multiple distance positions for fitting, so as to realize the calibration of the modeling parameters corresponding to the same column of each frame of the calibration plate image.
[0099] In summary, this embodiment does not need to convert into an intrinsic parameter matrix according to the intrinsic parameter calibration method disclosed in the prior art, and does not need to obtain the actual space coordinates through the light plane parameters and the camera intrinsic parameters like the traditional method, but relies on the motor to drive the calibration plate to slide to different distances to collect the light point pixel coordinates in the same column and their corresponding test distances, and fit the two corresponding modeling parameters in the same column. Moreover, these are two modeling parameters fitted by multiple frames of calibration plate images in the same column, rather than multiple optical parameters corresponding to multiple frames of calibration plate images in the same column. Compared with the prior art, the types and quantities of camera intrinsic parameters that need to be calculated are reduced, thereby simplifying the calculation amount of the transformation relationship from the camera coordinate system to the pixel coordinate system, and improving the calibration efficiency of the structured light module; when it is subsequently actually applied to ranging, the calculation efficiency is also improved.
[0100] On the basis of the above embodiment, every time the calibration of a column of modeling parameters is completed by executing step A2, step A3 is executed once. In step A3, i=i+1 is configured so that the pixels of the i+1th column in the jth frame calibration plate image are updated to the pixels of the ith column in the jth frame calibration plate image, and steps 21 to 24 are repeated until the N columns in each frame of the calibration plate image collected by step A1 are traversed, and the N columns of light point pixel coordinates and their corresponding test distances are determined to be obtained after nonlinear fitting, and N modeling parameter pairs are accumulated to determine that the calibration of the modeling parameters of the structured light module is completed; wherein, i in step 21 is incremented from the value 1, therefore, step 3 configuration i=i+1 is equivalent to step 21 traversing the next column of the same frame calibration plate image. Specifically, in M frames of calibration plate images, by repeatedly executing steps 21 to 24, N structured light ranging relational expressions are cumulatively fitted; N modeling parameter pairs correspond to N columns in each frame of calibration plate images, N columns are N columns calibrated in the pixel plane of the camera in the same frame of calibration plate images, and each column corresponds to a measurement angle and a test distance; i represents the column number of the currently traversed column in a frame of calibration plate images, i is a positive integer, i is greater than 0, and i is less than or equal to N. Therefore, there are N columns of pixel points or N calibration columns of light point pixel coordinates in the camera image space, and the methods of the aforementioned steps A2 and A3 can be used to perform simple modeling at N measurement angles in turn, and N structured light ranging relational expressions and modeling parameter pairs are cumulatively fitted, that is, 2*N modeling parameters are cumulatively fitted, and the camera calibration is completed. Therefore, this embodiment repeatedly executes step 21 to step 24 to complete the calibration of the corresponding modeling parameters column by column in each frame of the calibration plate image, thereby completing the calibration of the structured light module from multiple measurement angles.
[0101] Preferably, the nonlinear fitting adopts the least squares method; specifically, the pixel coordinates of the light spots in the i-th column of the j-th frame calibration plate image and the test distance corresponding to the i-th column of the j-th frame calibration plate image support fitting processing using the least squares method, and the mapping relationship corresponding to the i-th column is fitted, so that the optimal coefficients are fitted by the least squares method as the aforementioned modeling parameters, so that the fitting error is minimized.
[0102] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable technicians familiar with the technical field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.
Claims
1. A distance measurement control method based on a structured light module, characterized in that: The structured light module includes a structured light emitter and a camera; The ranging control method comprises: Step A: fitting modeling parameters using pixel coordinates and distance information, and constructing a distance measurement function using the modeling parameters as known quantities; Step B, controlling the structured light emitter to emit structured light to the surface of the object to be measured to form a light strip, and controlling the camera to collect an image of the structured light projected onto the surface of the object to be measured; Step C, using a light stripe center extraction algorithm to calculate the light stripe center pixel coordinates from the image captured by the structured light module; Step D: input the calculated light strip center pixel coordinates into the distance measurement function to calculate the distance between the measured object and the structured light module.
2. The ranging control method according to claim 1, characterized in that: After the light strip is imaged by the camera, it is represented by a plurality of corresponding columns of pixels in the pixel plane of the camera; each column of pixels forms a corresponding measurement angle relative to the optical axis of the camera; each column of pixels is used to identify the pixel position formed by the light spot of the light strip at each measurement angle in the photosensitive element of the camera; each column of pixels is fitted with corresponding modeling parameters in step A, and the ranging function constructed for each column of pixels in step A corresponds to a measurement angle.
3. The ranging control method according to claim 2, characterized in that: Whenever step C calculates the light strip center pixel coordinates at a measurement angle, step D calculates the distance between the measured object and the structured light module at the corresponding measurement angle, representing the distance information at the orientation of the corresponding measurement angle; wherein, there is at least one light strip center pixel coordinate in a column of pixel points, so that when step D uses the ranging function for conversion, the light spot pixel coordinates are configured to be represented by the light strip center pixel coordinates.
4. The ranging control method according to claim 2, characterized in that: In step C, the light stripe center extraction algorithm is used to extract the light stripe center line in the image captured by the structured light module, and multiple light stripe center pixel coordinates are obtained to form a center line point set; in the image including the light stripe, a column of pixels has at least one light stripe center pixel coordinate; Wherein, the light bar center extraction algorithm includes but is not limited to a grayscale centroid extraction algorithm.
5. The ranging control method according to claim 2, characterized in that: In step A, the method of fitting modeling parameters using pixel coordinates and distance information includes: Step A1, controlling the structured light emitter to emit structured light, projecting it onto the calibration plate to form a light strip; at the same time, controlling the calibration plate to move relative to the structured light module, and the camera collects calibration plate images at different test distances to obtain multiple frames of calibration plate images, and extracts the pixel coordinates of the light points in each column from the same frame of the calibration plate image, and determines the measurement distance corresponding to the same column of the same frame of the calibration plate image; Step A2, in the multiple frames of calibration plate images obtained in step A1, nonlinear fitting is performed on the light spot pixel coordinates and the corresponding test distances in the same column to obtain the modeling parameters corresponding to the same column; Step A3, repeatedly executing step A2 to traverse each column of light spot pixel coordinates and their corresponding test distances in each frame of the calibration plate image obtained in step A1, sequentially obtain the modeling parameters corresponding to all columns, and determine that the calibration of the modeling parameters of the structured light module is completed; The modeling parameters corresponding to the same column are the coefficients required to construct the structured light ranging relationship under the same column. The structured light ranging relationship is a relationship between the pixel coordinates of the light spots in the same column and their corresponding test distances constructed based on the pinhole imaging principle, so that the structured light ranging relationship under the same column forms the function to be fitted corresponding to the nonlinear fitting of step A2.
6. The distance measurement control method according to claim 5, characterized in that: Before executing step A1, the method includes: The emission direction of the structured light emitter is adjusted to form an angle with the optical axis direction of the camera, and the line between the center of the structured light emitter and the center of the camera is set to be parallel to the measured surface of the calibration plate; the emission direction of the structured light emitter is set to intersect with the calibration plate.
7. The ranging control method according to claim 6, characterized in that: In the step A1, the method of controlling the displacement of the calibration plate relative to the structured light module includes: driving the calibration plate to slide on the slide rail; each time the calibration plate slides a certain distance along the slide rail, the camera collects an image of the calibration plate, the light bar shifts once on the calibration plate, and the camera detects the pixel coordinates of the light point formed by the light bar in the pixel plane in real time, and records the test distance at the corresponding measurement angle; The relative position between the structured light emitter and the camera is fixed by a fixture.
8. The distance measurement control method according to claim 7, characterized in that: In the step A1, during the sliding of the calibration plate along the slide rail, each time the calibration plate slides a preset fixed distance, the camera captures an image of the light bar, the imaging position of the light bar in the imaging plane is offset by a preset height in the height direction, and the pixel coordinates of the light point at the corresponding measurement angle are offset by a preset longitudinal offset in the opposite direction of the height direction in the pixel plane; Wherein, the offset formed by the imaging position of the light strip in the imaging plane at the same measuring angle relative to the reference position at the same measuring angle is marked as the light strip offset at the same measuring angle; The reference position is the imaging position of the reflected light of the structured light emitted by the structured light emitter on the measured surface of the measured object farthest from the structured light module in the detectable area of the structured light module in the camera; the measured object includes the calibration plate.
9. The ranging control method according to claim 8, characterized in that: In the same column, the methods for constructing the structured light ranging relationship include: The line between the center of the structured light emitter and the optical center of the camera is set as the baseline, and the length of the baseline is set as the baseline distance; The distance between the baseline at the corresponding measurement angle and the calibration plate is set as the test distance d at the corresponding measurement angle; The structured light ranging relationship corresponding to the measurement angle is: Px=k0 / d+k1, forming a relationship between the pixel coordinates of the light point in the direction represented by a column of pixels and the test distance; Where Px is the pixel coordinate of the light spot at the corresponding measurement angle; Among them, the modeling parameters corresponding to the same column include k0 and k1; At the same measurement angle, the ratio of the baseline distance to the test distance is equal to the ratio of the light bar offset to the focal length of the camera; At the same measurement angle, the sum of the product of the light spot pixel coordinates and the single pixel size and the projection error is equal to the light bar offset; the single pixel size and the projection error are both predetermined; the imaging of the light bar in the imaging plane of the camera is set in the imaging plane coordinate system, wherein the reference position and the light bar offset are both set in the imaging plane coordinate system; the light spot pixel coordinates are set in the pixel coordinate system, and the pixel coordinate system is the coordinate system of the pixel plane of the camera.
10. The distance measurement control method according to claim 9, characterized in that: In step A, the method of constructing a ranging function with modeling parameters as known quantities includes: When it is determined that the structured light ranging relationship at the corresponding measurement angle is Px=k0 / d+k1, and the modeling parameters k0 and k1 at the corresponding measurement angle are obtained through step A3, the light spot pixel coordinates Px at the corresponding measurement angle and the test distance d at the corresponding measurement angle are set to unknown quantities; then Px=k0 / d+k1 is converted to d=k0 / (Px-k1), and the converted d=k0 / (Px-k1) is recorded as the ranging function at the corresponding measurement angle.
11. The distance measurement control method according to claim 9, characterized in that: In step A2, the method of performing nonlinear fitting on the light spot pixel coordinates in the same column and the corresponding test distances includes: Step 21, sequentially traverse the i-th column of the M frames of calibration plate images acquired in step A1; wherein the i-th column is a column of pixel positions projected by the light spot of the light strip at the corresponding measurement angle in the photosensitive element of the camera, and is recorded as a calibration column; Step 22, extract the row pixel position coordinate Pxj of the center of the light strip in the i-th column from the currently traversed frame of the calibration plate image, and record it as the pixel coordinate of the light spot in the i-th column of the currently traversed frame of the calibration plate image, and at the same time determine the test distance dj corresponding to the i-th column of the same frame of the calibration plate image; j represents the sorting number of the currently traversed frame of the calibration plate image, so that the currently traversed frame of the calibration plate image is marked as the j-th frame of the calibration plate image; j is a positive integer, j is greater than 0, and j is less than or equal to M; Step 23, configure j=j+1, so that the data of the j+1th calibration plate image is updated to the data of the jth calibration plate image, and then repeat step 22 until the i-th column of the M-frame calibration plate image is traversed, and the i-th column of light point pixel coordinates in each frame of the calibration plate image are combined into a coordinate set {Px1, Px2, ..., PxM}, and at the same time, the test distance corresponding to the i-th column in each frame of the calibration plate image is combined into a distance set {d1, d2, ..., dM} or a distance set {1 / d1, 1 / d2, ..., 1 / dM}; Step 24, configure the coordinate set of step 23 and the distance set of step 23 as data sets on two coordinate axes, perform nonlinear fitting, obtain the corresponding modeling parameters k0 and modeling parameters k1 of the i-th column, and determine the fitted structured light ranging relationship corresponding to the i-th column; and record the modeling parameters k0 and modeling parameters k1 corresponding to the i-th column as a modeling parameter pair of the i-th column; then execute step A3.
12. The distance measurement control method according to claim 11, characterized in that: In step A3, i=i+1 is configured so that the pixels in the i+1th column in the jth frame calibration plate image are updated to the pixels in the ith column in the jth frame calibration plate image, and steps 21 to 24 are repeated until the N columns in each frame calibration plate image acquired in step A1 are traversed, and after determining that the N columns pass the nonlinear fitting, N modeling parameter pairs are accumulated, and it is determined that the calibration of the modeling parameters of the structured light module is completed; In the M frames of calibration plate images, by repeatedly executing steps 21 to 24, N structured light ranging relationship equations are cumulatively fitted; the N modeling parameter pairs correspond to the N columns calibrated in each frame of the calibration plate image, and each column corresponds to one of the measurement angles and one of the test distances; i represents the column number of the currently traversed column in a frame of the calibration plate image, i is a positive integer, i is greater than 0, i is less than or equal to N.
13. The distance measurement control method according to claim 12, characterized in that: Non-linear fitting includes, but is not limited to, the least squares method.
Citation Information
Cited By
Embroidery machine needle eye positioning method based on image analysis
CN122329151A
