Modeling parameter calibration method of structured light module, chip and calibration device

By controlling the displacement of the calibration plate and acquiring images in the structured light module, and using nonlinear fitting to construct a ranging relationship, the problem of low camera calibration efficiency in structured light vision systems is solved, and efficient structured light module calibration and ranging are achieved.

CN119027509AActive Publication Date: 2024-11-26AMICRO SEMICONDUCTOR CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202310605822.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-26
Publication Date
2024-11-26
Estimated Expiration
2043-05-26

Smart Images

  • Figure CN119027509A_ABST
    Figure CN119027509A_ABST
Patent Text Reader

Abstract

The invention discloses a modeling parameter calibration method of a structured light module, a chip and a calibration device, and the method comprises the following steps: 1, controlling a structured light emitter to emit structured light, and projecting the structured light to a calibration plate to form a light strip; controlling the calibration plate to displace relative to the structured light module, obtaining multiple frames of calibration plate images, extracting light bar center coordinates of each calibration column from the same frame of calibration plate image, and determining a corresponding test distance at the same time; 2, in the obtained multiple frames of calibration plate images, nonlinear fitting is carried out on the light strip center coordinates of the same calibration column and the test distance corresponding to the same calibration column, and modeling parameters corresponding to the same calibration column are obtained; and step 3, sequentially obtaining modeling parameters corresponding to all calibration columns by repeatedly executing the step 2 to obtain the light stripe center coordinates of each calibration column in each frame of calibration plate image and the corresponding test distance. And the calibration efficiency of the structured light module is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of camera calibration, and in particular to a calibration method, chip and calibration device for modeling parameters of a structured light module. Background Art

[0002] The structured light vision system is a measurement system that combines a line laser and a monocular camera. For example, a line laser sensor is composed of a camera and a line laser. Before using the structured light vision system, camera calibration and laser plane calibration are required to achieve three-dimensional measurement through imaging principles and laser plane parameters. This calibration method is applicable to most cameras. The line laser projects laser light onto the surface of the object to be measured. The laser light enters the receiving lens of the camera after diffuse reflection and forms an image on the camera's photosensitive element. The position of the image changes with the distance of the target. By imaging on the camera's photosensitive element and solving the relative position of the image, the relative height of the object to be measured can be obtained. However, a special calibration target is required to calibrate the camera's internal and external parameters at the same time, which increases the complexity of the calibration calculation process and the difficulty of fitting, resulting in a relatively low camera calibration efficiency. Summary of the invention

[0003] This application discloses a modeling parameter calibration method, chip and calibration device for a structured light module. The specific technical solution is as follows: The modeling parameter calibration method of the structured light module includes a structured light emitter and a camera; the modeling parameter calibration method includes the following steps: step 1, controlling the structured light emitter to emit structured light, projecting it onto a calibration plate to form light strips; at the same time, controlling the calibration plate to move relative to the structured light module, and the camera collects calibration plate images at different test distances to obtain multiple frames of calibration plate images, and extracts the center coordinates of the light strips of each calibration column from the same frame of the calibration plate image, and at the same time determines the test distance corresponding to the same calibration column of the same frame of the calibration plate image; each frame of the calibration plate image includes the light strips; step 2, in the multiple frames of the calibration plate images obtained in step 1, nonlinearly calibrates the center coordinates of the light strips of the same calibration column and the test distance corresponding to the same calibration column The method comprises the following steps: performing a property fitting to obtain the modeling parameters corresponding to the same calibration column; step 3, traversing the light bar center coordinates and the corresponding test distances of each calibration column in each frame of the calibration plate image obtained in step 1 by repeatedly executing step 2, obtaining the modeling parameters corresponding to all calibration columns in turn, and determining that the calibration of the modeling parameters of the structured light module is completed; wherein, the number of executions of step 2 is equal to the number of calibration columns in one frame of the calibration plate image, and the number of calibration columns in each frame of the calibration plate image is equal; the modeling parameters corresponding to the same calibration column are the coefficients required to construct the structured light ranging relationship under the same calibration column, and the structured light ranging relationship is the relationship between the light bar center coordinates of the same calibration column and the corresponding test distance constructed based on the pinhole imaging principle.

[0004] In summary, steps 1 to 3 disclosed in this embodiment rely on controlling the displacement of the calibration plate relative to the structured light module, and using the center coordinates of the light bars of different calibration columns in the calibration plate images collected at different distances and their corresponding test distances to perform nonlinear fitting to complete parameter calibration, so that the fitted modeling parameters can replace the traditional camera intrinsic parameters in a direction or a test angle relative to the optical axis of the camera, thereby ensuring the calibration accuracy of the modeling parameters, that is, ensuring the accuracy of the calibration parameters of the structured light module, and improving the efficiency of the calibration of the structured light module. In addition, based on the modeling parameters corresponding to each calibration column fitted by this application and the structured light ranging relationship that can be constructed, the ranging modeling method between the structured light and the camera or the calibration method of the camera is simplified to only fitting and calculating the modeling parameters corresponding to each calibration column, then the calibration of the camera intrinsic parameters is replaced or omitted before the ranging operation, so that the camera calibration calculation amount is reduced, and the calibration efficiency of the structured light module is improved; further, after obtaining the modeling parameters, the modeling and ranging operations can be performed without calibrating the camera intrinsic parameters, and the calculation efficiency is improved.

[0005] A chip is used to execute the modeling parameter calibration method, serving as the execution body of step 1, step 2 and step 3 in the aforementioned embodiment.

[0006] A calibration device comprises a calibration plate, a slide rail and the chip, and the calibration device is used to calibrate the structured light module, wherein the chip is used to control the calibration plate to slide along the slide rail, and control the structured light emitter to emit structured light to the measured surface of the calibration plate, and at the same time control the camera to collect the image of the structured light projected onto the measured surface of the calibration plate, so as to obtain the modeling parameters corresponding to each calibration column in each frame of the calibration plate image by executing steps 1 to 3, and determine that the calibration of the modeling parameters of the structured light module is completed. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is a flowchart of a method for calibrating modeling parameters of a structured light module disclosed in one embodiment of the present application.

[0008] Figure 2 This is a schematic diagram of the optical path principle of a single-point line laser disclosed in one embodiment of the present application.

[0009] Figure 3 This is a schematic diagram of the relative positions of a structured light emitter, a camera and an object to be measured disclosed in an embodiment of the present application. Implementation

[0010] The technical scheme in the embodiment of the present invention will be described in detail below in conjunction with the drawings in the embodiment of the present invention. To further illustrate each embodiment, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments, and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in this field should be able to understand other possible implementations and advantages of the present invention. It can be understood by ordinary technicians in related fields that the goal of camera calibration is to find a suitable or approximate mathematical model, find the parameters of this model, and use a simple mathematical model to express a complex imaging process.

[0011] Considering that in practical applications, the structured light vision system uses a small-angle camera that meets the pinhole imaging principle, and there is a certain angle between the forward emission direction of the straight line laser and the optical axis of the camera, and in the optical path system, the line laser is mostly oblique relative to the surface of the incident object. In order to reduce the complexity of calibration and improve the calibration efficiency, this application aims to provide a fast calibration method that does not rely on camera intrinsic calibration for a structured light module composed of a small-angle distortion-free camera that meets the pinhole model, and uses the light strip image data on the calibration plate at different distances to fit the parameters required for the distance measurement function relationship of the structured light module, so as to simplify the calibration steps of the camera intrinsic parameters and the camera extrinsic parameters.

[0012] As an embodiment, a method for calibrating modeling parameters of a structured light module is disclosed. The execution subject of the method for calibrating modeling parameters can select a controller built into the structured light module or an external control device. The structured light module includes a structured light emitter and a camera. Figure 1 As shown, the modeling parameter calibration method includes the following steps: Step 1: Control the structured light emitter to emit structured light, project it onto the calibration board and form a light strip, for example, Figure 3 The structured light emitted by the structured light emitter in the structured light emitter (considered to be composed of multiple single-point line lasers with continuous angles, which are Figure 3The outgoing light 301, the outgoing light 302, the outgoing light 303 and the like are laser lights emitted from the light source of the structured light emitter), and the optical path is changed through the lens and the wave mirror of the camera, so that the structured light emitted by the structured light emitter forms a structured light surface, which can be understood as the effect that the emitted light is an approximate plane; the structured light surface intersects with the surface of the object to be measured to form the light strip shown in the figure, and the camera is controlled to collect the calibration plate image, including the original image of the light strip, wherein a single-point line laser can be regarded as emitted by a single-point line laser emitter. For example, a single-point line laser is used to emit a single-point line laser, and multiple single-point line lasers are used to emit multiple single-point line lasers, then a line laser A laser plane can be emitted, and the emitted laser plane can be regarded as composed of multiple single-point line lasers, then the straight line laser can be equivalently decomposed into multiple single-point line lasers; similarly, the structured light emitter disclosed in the present application can be regarded as composed of multiple single-point line laser emitters, each single-point line laser emitter emits a single-point line laser, and multiple single-point line lasers form a structured light surface, that is, the structured light emitter emits a structured light surface and intersects with the calibration plate to form a light strip. When using single-point laser triangulation ranging, that is, using the pinhole imaging principle (pinhole model) to establish a mathematical model or range measurement, it is necessary to decompose the structured light into multiple single-point line lasers to correspond to the optical path of the single-point line laser.

[0013] Furthermore, in step 1, the calibration plate is also controlled to move relative to the structured light module, and the calibration plate can be driven by a motor to move relative to the camera, so that the camera obtains images of the calibration plate at different distances, and each frame of the calibration plate image includes a light bar. Generally, the calibration plate moves away from the camera or close to the camera in one direction.

[0014] The camera collects calibration plate images at different test distances to obtain multiple frames of calibration plate images. Each frame of the calibration plate image is, in turn, an image of the calibration plate at different distances relative to the structured light module at the same position, and all include light strips, that is, each frame of the calibration plate image includes imaging information of the light strips. And for each frame of calibration plate image collected, the number of calibration columns set in the pixel plane of the camera for each frame of calibration plate image is equal; each time the camera collects one or more frames of calibration plate image, the center coordinates of the light strips of each calibration column are extracted from the same frame of calibration plate image, and the test distance corresponding to the same calibration column of the same frame of calibration plate image is determined, and the test distance can be understood as the distance between the light strip and the structured light module at the angle or direction corresponding to the calibration column; in step 1, the structured light module can actively or under the command of an external controller extract height calibration data, X-axis component or Y-axis component of the light strip point cloud, X-direction calibration data of the black and white area or white area in the calibration plate, etc. from the calibration plate image, which can be used to convert the row pixel position information of the light strip (or the center line of the light strip), or extract the center of gravity position of each column of the light strip, including the center coordinates of the light strip, to provide input information for subsequent fitting relationship and calculation modeling parameters.

[0015] In some embodiments, before the calibration plate is displaced, a certain distance is left between the calibration plate and the structured light module, so that when the calibration plate is displaced relative to the structured light module, the camera can collect two-dimensional images of the calibration plate at multiple distances, establish a more comprehensive calibration plate image library, and obtain pixel position information at more distances, including the center coordinates of the light strips corresponding to the light strips at different distances. The formula required to construct the pinhole imaging principle can be fitted to obtain the corresponding optimal fitting parameters, which can replace the parameters of the camera intrinsic parameters and simplify the camera calibration method.

[0016] Step 2. In the multiple frames of calibration plate images obtained in step 1, nonlinear fitting is performed on the center coordinates of the light strips in the same calibration column and the test distance corresponding to the same calibration column to obtain the modeling parameters corresponding to the same calibration column and reduce the fitting error; wherein the same calibration column is a column with the same order from each frame of calibration plate images, which represents the projection position information of the light strips in the pixel position, specifically, the pixel coordinates of the light strips projected at an angle or direction in each frame of calibration plate images, and then nonlinear fitting is performed on the test distances corresponding to the same calibration column of the multiple frames of calibration plate images to obtain the coefficients required for constructing the structured light ranging relationship under the pinhole imaging principle, so as to minimize the fitting error generated on the same calibration column in each frame of calibration plate images and ensure the fitting accuracy, which is equivalent to ensuring the accuracy of the camera's intrinsic parameters.

[0017] In this embodiment, the modeling parameters corresponding to the same calibration column are the coefficients required to construct the structured light ranging relationship under the same calibration column. The structured light ranging relationship is a relationship between the center coordinates of the light strips of the same calibration column and the corresponding test distances constructed based on the pinhole imaging principle. Constructing / defining the structured light ranging relationship under the pinhole imaging principle can be specifically understood as a relationship between the center coordinates of the light strips constructed under the pinhole imaging principle and the corresponding test distances. The coefficients required to construct the structured light ranging relationship under the same calibration column can be fitted by combining the data of the calibration plate image at different distances. In this embodiment, the structured light ranging relationship can be used as the corresponding function to be fitted in the nonlinear fitting to fit the modeling parameters, including at least two fitting coefficients. Therefore, compared with the line laser-camera modeling ranging scene in the prior art, this embodiment converts the mathematical formula constructed by the triangle ranging into the relationship between the center coordinates of the light strips and the corresponding test distances, without the need to additionally calibrate the internal and external parameters of the camera, and the calibration calculation amount is less.

[0018] In this embodiment, during the process of performing nonlinear fitting on the center coordinates of the light stripes of the same calibration column and the test distance corresponding to the same calibration column, the optimal relationship between the center coordinates of the light stripes and the test distance is fitted and the modeling parameters are obtained, wherein the center coordinates of the light stripes and the test distance (the distance of the calibration plate relative to the structured light module) extracted from the calibration plate image at different distances are known quantities that can be acquired in real time from each frame of the calibration plate image. Therefore, these known quantities can be used to fit and calculate the coefficients required to construct the structured light ranging relationship from the image data of multiple frames of the calibration plate image, thereby establishing a mathematical fitting model for the structured light module ranging.

[0019] It should be noted that the test distance is the distance between the calibration plate and the structured light module, specifically the distance between the line between the optical center of the camera and the center of the structured light emitter and the calibration plate (the measured surface of the calibration plate), which is regarded as the distance measurement distance of the structured light module. The structured light emitted by the structured light emitter is a straight line structured light, which is projected onto the surface of the measured object to form Figure 3 The straight line of light shown. The straight line structured light can be approximately decomposed into multiple single point line lasers, which can be multiple single point line lasers with continuous angles. A single point line laser can be regarded as a straight line light. If the structured light emitter is decomposed into multiple single point line laser emitters, the test distance is configured as the distance between the line connecting the center of the single point line laser emitter and the optical center of the camera and the calibration plate. For the light strip, Figure 3In the two-dimensional light strip image of the intersection of the structured light surface and the surface of the object to be measured, the center coordinates of the light strip can be extracted, including the X-axis component and the Y-axis component, which correspond to the U-axis component (expressed by the number of pixels) and the V-axis component (expressed by the number of pixels) of the pixel coordinate system, which are derived from the linear structured light or the result of the laser projection of multiple single-point lines. In this embodiment, the center coordinates of the light strip can represent the pixel coordinates that are not affected by distortion.

[0020] Therefore, in step 2, the aforementioned modeling parameters can be calculated by performing nonlinear fitting on the mapping relationship between the center coordinates of the light stripes and their test distances in the same calibration column of multiple frames of calibration plate images, as the optimal parameters for constructing the mapping relationship between the center coordinates of the light stripes and their test distances in the same calibration column, that is, the coefficients required to construct the structured light ranging relationship. The fitted modeling parameters can replace the traditional camera intrinsic parameters in a direction or a test angle relative to the optical axis of the camera, so as to replace or omit the calibration of the camera intrinsic parameters before the ranging operation.

[0021] Step 3, by repeatedly executing step 2, the light bar center coordinates and the corresponding test distance of each calibration column in each frame of the calibration plate image obtained in step 1 are traversed, the modeling parameters corresponding to all calibration columns are obtained in turn, and the calibration of the modeling parameters of the structured light module is determined to be completed; wherein, the number of executions of step 2 is equal to the number of calibration columns in a frame of the calibration plate image, and the number of calibration columns in each frame of the calibration plate image is equal. In this way, step 3 can traverse all frames of calibration plate images in turn by repeatedly executing step 2 to obtain each calibration column in the same frame of the calibration plate image and determine the corresponding test distance on the calibration column, and can also complete the traversal of each calibration column (or all pixel columns) in the same frame of the calibration plate image to extract the light bar center coordinates in each calibration column. In order to obtain the modeling parameters corresponding to the same calibration column, it is necessary to first traverse the light bar center coordinates in the same calibration column of all frames of the calibration plate image, and then traverse the light bar center coordinates in the next calibration column of all frames of the calibration plate image. Based on this traversal order, the modeling parameters corresponding to all calibration columns are obtained in turn through the nonlinear fitting of step 2, and the calibration of the modeling parameters of the structured light module is determined to be completed, including the calibration of the modeling parameters at all test angles or all calibration columns, thereby determining the completion of the calibration of the structured light module.

[0022] In summary, steps 1 to 3 disclosed in this embodiment rely on controlling the displacement of the calibration plate relative to the structured light module, and performing nonlinear fitting using the center coordinates of light bars in different calibration columns in the calibration plate image collected at different distances and their corresponding test distances to complete parameter calibration. The fitted modeling parameters can replace the traditional camera intrinsic parameters in a direction or a test angle relative to the optical axis of the camera, thereby improving the efficiency of the structured light module calibration while ensuring the accuracy of the calibration parameters of the structured light module.

[0023] In addition, based on the modeling parameters corresponding to each calibration column fitted by the present application and the structured light ranging relationship that can be constructed, the ranging modeling method between the structured light and the camera or the camera calibration method is simplified to just fitting and calculating the modeling parameters corresponding to each calibration column. This replaces or omits the calibration of the camera intrinsic parameters before the ranging operation, reduces the amount of camera calibration calculation, and improves the calibration efficiency of the structured light module. Furthermore, after obtaining the modeling parameters, modeling and ranging operations can be performed without calibrating the camera intrinsic parameters, thereby improving the calculation efficiency.

[0024] As an embodiment, after the light strip is imaged by the camera, it is represented by corresponding multiple columns of pixel points in the pixel plane (i.e., multiple calibration columns of pixel points), each column of pixel points identifies the pixel position of each point cloud point constituting the light strip projected in the camera's photosensitive element, and each column of pixel points has the center coordinates of the light strip in the coordinates in the pixel coordinate system. In this embodiment, the camera can detect in real time the center coordinates of the light strip in each column formed by the projection of the light strip in the pixel plane. Each light strip contained in the calibration plate image can be represented by multiple columns of pixel points (i.e., multiple calibration columns of pixel points) in the pixel coordinate system. Although the measured surface projected onto the measured object or the measured surface of the calibration plate is Figure 3 The straight line light shown is in the outgoing light of the structured light emitter (the angle range formed by the single-point line laser 303 and the single-point line laser 301), and the corresponding light paths are all within the visible range of the camera. However, the number of calibration columns formed by the light strips and the distribution of calibration columns projected by the same light strip on the pixel plane can be determined according to the camera resolution of the structured light module, the operating computing power of the controller of the structured light module, and the specifications of the internal and external parameters of the camera, and this embodiment does not impose any restrictions.

[0025] It can be understood that the structured light emitted by the structured light emitter changes the optical path through the camera so that the structured light forms a structured light surface. The structured light surface can be equivalent to being decomposed into multiple single-point line lasers according to the test angle; the light source of the structured light emitter has only one point, and the optical path is changed by the lens and the wave mirror, so that the emitted structured light forms an effect of an approximate plane, which is equivalent to a structured light surface, such as Figure 3The structured light surface shown; for example, when the structured light is a line laser, the line laser can be regarded as a collection of multiple single-point line lasers, rather than being composed of multiple single-point line lasers in a physical sense. In the structured light surface, a single-point line laser forms a corresponding test angle relative to the optical axis of the camera, and a single-point line laser forms the center coordinate of the light bar in a calibration column of the pixel plane of the camera; the light bar is the intersection of the structured light surface and the measured surface of the calibration plate. In order to form the light bar, the calibration plate here can also be the measured object, and the measured surface of the calibration plate is the measured surface of the measured object.

[0026] In this embodiment, the pixel positions of the light spots of the light bar at various test angles projected in the photosensitive element of the camera are identified as pixel positions in the corresponding calibration columns of the pixel plane, so that the light bar is represented by pixel points in multiple calibration columns in the pixel plane; each calibration column corresponds to a test angle, that is, the pixel points in each calibration column also correspond to the light spots of the light bar at the corresponding test angle. Each column of pixels forms a corresponding test angle relative to the optical axis of the camera. There is a light bar center coordinate in the pixel coordinates of a column of pixels projected by the light bar, that is, there is a light bar center coordinate in the pixel coordinates of a calibration column. A light bar center coordinate corresponds to a test angle. A test angle represents the azimuth formed by a position point on the measured surface of the calibration plate relative to the optical axis of the camera. The structured light emitted by the structured light emitter changes the optical path after passing through the camera to form a structured light surface, which can be regarded as composed of multiple single-point line lasers on the basis of determining multiple test angles; within the structured light surface, a single-point line laser forms a corresponding test angle relative to the optical axis of the camera, and a single-point line laser is correspondingly projected in a calibration column of the pixel plane (pixel coordinate system) of the camera to form the light bar center coordinate, so that each pixel point in the calibration column is located at a pixel position formed by the projection of a single-point line laser in the pixel plane after being reflected by the calibration plate. Since the structured light surface can be approximately regarded as a combination of countless single-point line lasers emitted from the structured light emitter, the center coordinates of the light strips in the same calibration column and the test distances corresponding to the same calibration column can be nonlinearly fitted according to step 2, so as to obtain corresponding modeling parameters at a finite number of test angles, and obtain the distance information of the measured object at a finite number of test angles of the camera, which can meet some simple contour point ranging application scenarios.

[0027] Preferably, since the structured light emitted by the structured light emitter can be regarded as forming a structured light surface, the light strip is located at the intersection of the structured light surface and the measured surface of the calibration plate, so the projection position of each laser point constituting the light strip in the photosensitive element of the camera can be connected into a horizontal projection line segment in the pixel coordinate system; thereby reflecting that the structured light is a straight-line structured light and the light strip is parallel to the slide rail, reducing the complexity of the posture conversion required for calibration and improving the efficiency of calibration.

[0028] On the basis of the above embodiment, since each calibration column of pixels identifies the pixel position of each light spot that constitutes the light strip projected into the photosensitive element of the camera, a column of pixels (i.e., the pixels in a calibration column) forms a corresponding test angle relative to the optical axis of the camera, a single-point line laser forms a corresponding test angle relative to the optical axis of the camera, and a single-point line laser is correspondingly projected within a column of the pixel plane of the camera to form the center coordinate of the light strip, so each column of pixels has at least one of the center coordinates of the light strip and is located at a corresponding test angle relative to the optical axis of the camera. Therefore, the center coordinates of the light strips in each calibration column (derived from the corresponding column of pixels) are obtained through nonlinear fitting in step 2 to obtain the corresponding modeling parameters, which are the coefficients required to construct the structured light ranging equation under the same calibration column, so that the structured light ranging equation under the same calibration column forms the function to be fitted corresponding to the nonlinear fitting of step 2; therefore, the structured light ranging equation under each calibration column corresponds to a test angle, so that: the structured light emitter is equivalent to being decomposed into a plurality of single-point line laser emitters, each single-point line laser emitter emits a single-point line laser, and the corresponding test distance at each test angle is updated to the distance between the line connecting the center of the single-point line laser emitter and the optical center of the camera and the calibration plate, therefore, steps 1 to 3 are executed in turn for the ranging system composed of the single-point line laser emitter and the camera at each test angle.

[0029] It should be noted that the modeling parameters corresponding to the same calibration column are the coefficients required to construct the structured light ranging relationship under the same calibration column. The structured light ranging relationship is the relationship between the center coordinates of the light strips in the same calibration column and their corresponding test distances constructed based on the pinhole imaging principle.

[0030] Based on the embodiments described in the aforementioned steps 1 to 3, when a camera that satisfies the pinhole imaging principle is used, each column of pixels on the pixel plane can be approximately regarded as a picture seen at the same vertical angle of the structured light surface. At this vertical angle, there is a single-point line laser emitter (similar to the function of a single-point laser) "extracted" from the straight-line structured light (the light strip), and the vertical angle is the aforementioned test angle. Therefore, by executing steps 1 to 3, it can be modeled with two modeling parameters (corresponding to constructing the structured light ranging relationship, and a structured light ranging relationship is constructed at a vertical angle). By analogy, if there are N columns of pixels in the pixel plane where the calibration plate image is located, N light strip center coordinates may be extracted therefrom; the ranging system composed of the single-point line laser emitter and the camera at each test angle is modeled in turn by executing steps 1 to 3, that is, the structured light ranging relationship is constructed at each test angle to obtain the corresponding modeling parameters, avoiding the need to calibrate the camera internal parameters first; and the ranging modeling of the entire structured light module is also completed.

[0031] As an embodiment, before executing step 1, the emission direction of the structured light emitter is configured to be set at an angle with the optical axis direction of the camera, and the line between the center of the structured light emitter and the optical center of the camera is set to be parallel to the measured surface of the calibration plate; the structured light emitted by the structured light emitter is set to intersect with the measured surface of the calibration plate, and the intersection line forms the light strip in the measured surface of the calibration plate, so that the structured light emitted by the structured light emitter is incident on the calibration plate at a certain angle to the normal of the measured surface of the calibration plate, and the reflected light is imaged into the photosensitive element of the camera through the lens of the camera; the positions of the calibration plate, the structured light emitter and the camera are set, including determining the relative position relationship between any two devices, and making device preparations for the calibration of the camera. The camera collects the image of the reflected light of the structured light in the photosensitive element of the camera, and uses the image of the light strip in the imaging plane of the camera to represent it; the image of the light strip in the imaging plane of the camera is a part of the calibration plate image, so that the calibration plate image forms a two-dimensional structured light stripe image. In order to reduce distortion calculations, the camera included in the structured light module uses a small-angle distortion-free camera to meet the pinhole imaging principle and ensure the reliability of the camera's internal parameters; the image sensor / photosensitive element set inside the camera can collect the reflected light of the structured light through an oblique optical path; the photosensitive element is a semiconductor device that converts light images into electronic signals; the distance between the surface of the photosensitive element and the optical center of the camera is equal to the focal length; the mass of photosensitive small molecules on the surface of the photosensitive element is called a pixel. Generally, the structured light module adopts the principle of triangular reflection, and the surface of the photosensitive element can be regarded as a matrix, so a light strip corresponding to the structured light can be projected, and the displacement of all light spots on the light strip in the calibration plate and the offset of the center coordinates of the light strip in the pixel plane can be measured at the same time.

[0032] It should be noted that the structured light emitter can be, but is not limited to, a light source that supports emitting linear structured light, such as a line laser emitter. The relative position between the structured light emitter and the camera is fixed, so that the structured light module is fixedly set at one position to form a sensor device with a fixed detection angle; an initial distance is set between the line between the center of the structured light emitter and the optical center of the camera and the calibration plate, which serves as the starting point for the displacement of the calibration plate relative to the structured light module. The line between the center of the structured light emitter and the optical center of the camera is also set as a fixed baseline, which can form a fixed assembly angle with the displacement direction of the calibration plate.

[0033] As an embodiment, before executing step 1, the emission direction of the structured light emitter is adjusted to be set at an angle with the optical axis direction of the camera, and the emission direction of the structured light emitter is set at a fixed angle with the optical axis direction of the camera, so that an oblique light path is formed when the calibration plate is used for calibration calculation; the line between the center of the structured light emitter and the center of the camera is set to be parallel to the measured surface of the calibration plate, which can be fixed by a special fixture. Note that it is not limited that the camera and the calibration plate must be parallel, nor is it limited that the intersection line of the structured light surface emitted by the structured light emitter and the calibration plate must be horizontal. Among them, the angle between the emission direction of the structured light emitter and the optical axis direction of the camera is fixed by the fixture, which is used to project structured light and collect images on the same calibration plate, regardless of whether the calibration plate is displaced; the relative position between the center of the structured light emitter and the center of the camera, and the angle between the emission direction of the structured light emitter and the optical axis direction of the camera are fed back to the controller of the structured light module, and the controller of the structured light module can fix the relative position and angle by controlling the fixture.

[0034] Combination Figure 2 and Figure 3 It can be seen that the controller of the structured light module sets the emission direction of the structured light emitter to intersect with the calibration plate so that the structured light emitter Figure 2 The structured light emitted (usually a single light ray, such as a single-point line laser, which can be referred to as the optical path of a single-point line laser) is incident on the calibration plate at a certain angle to the normal of the measured surface of the calibration plate, and the reflected light is imaged into the camera's photosensitive element through the camera lens, and the camera collects the image formed by the reflected light, wherein the camera collects the image of the reflected light of the structured light in the camera's photosensitive element, and uses the image of the light bar in the camera's imaging plane to represent it, and the image of the light bar in the camera's imaging plane is part of the calibration plate image. When the structured light emitter adopts a line laser emitter, the emitted laser line is structured light, such as a line laser, which can be regarded as a laser plane composed of countless laser lines emitted by the same line laser emitter, corresponding to Figure 3 The intersection of the laser plane and the calibration plate forms a light strip, so that the light strip is a line structured light strip, such as a projection spot of a straight line laser in the calibration plate; it can be extracted from the calibration plate image to perform a triangulation operation (constructing similar triangles for distance calculation), specifically, each pixel point on the light strip (the projection position of each single-point line laser in the pixel plane or the imaging position in the imaging plane) can perform a triangulation operation (using the aforementioned modeling parameters to construct a structured light ranging relationship for triangulation ranging calculation), and obtain the distance information of each pixel position within the angle range covered by the light strip relative to the camera. In this way, the relative posture between the camera and the structured light emitter is kept unchanged, and similar triangles are stably established for triangulation ranging calculation.

[0035] It should be noted that the above description is only for the convenience of explaining the relative positions of the components in the structured light module. When the relative positions remain unchanged, the structured light module and the calibration plate do not necessarily have to be placed on the horizontal plane, but can also be placed on the vertical plane. When executing step 1, driven by the motor, the calibration plate moves freely or rotates relative to the slide rail, and the placement posture is unconstrained.

[0036] On the basis of the above embodiments, when the surface of the photosensitive element of the camera (photosensitive surface) is preferably parallel to the measured surface of the calibration plate, the object plane (from the calibration plate) and the image plane (photosensitive element of the camera) determined by the pinhole imaging principle are kept parallel, which is convenient for establishing similar triangles for triangular distance measurement calculation, thereby utilizing the triangulation distance measurement method commonly used in single-point line laser distance measurement combined with a nonlinear fitting method (such as the least squares method) to accelerate the calibration of modeling parameters. At this time, it may be necessary to decompose the structured light surface into multiple single-point line lasers for processing. For example, the structured light (straight line light) is divided into single-point line lasers in multiple directions according to a preset angle, and then the triangulation distance measurement method (pinhole imaging principle) and nonlinear fitting are used for each single-point line laser to accelerate the calibration of the pixel coordinates of the same calibration column, so that the corresponding modeling parameters are calibrated in different directions.

[0037] As an embodiment, in step 1, the method for controlling the displacement of the calibration plate relative to the structured light module includes: the controller of the structured light module can drive the calibration plate to slide on the slide rail through a motor; under the drive of the motor, the camera collects the image of the calibration plate once every time the calibration plate slides a certain distance along the slide rail, and the light bar is offset once on the calibration plate, and the calibration plate is offset along the height direction, and the offset distance can be detected by the structured light module in real time; the line between the center of the structured light emitter and the optical center of the camera is set as the baseline; the relative position between the structured light emitter and the camera is fixed, the structured light module is fixedly installed at one end of the slide rail, and the initial position of the calibration plate is set at the other end of the slide rail, wherein the calibration plate is slidably set on the slide rail. Each time the calibration plate image is collected, the calibration plate is located at a different position on the slide rail. When the structured light emitter is turned on, the camera collects an image containing the light strip. The camera detects the center coordinates of the light strip formed by the projection of the light strip in the pixel plane in real time, and extracts the center coordinates of the light strip formed by the light strip in the pixel plane from the calibration plate image, and records the test distance at the corresponding test angle for participating in the nonlinear fitting of step 2.

[0038] Specifically, when the calibration plate is set on the normal plane of the slide rail, the height direction of the calibration plate can be regarded as the vertical direction of the slide rail, or parallel to the normal plane of the slide rail, and can also be used as the longitudinal direction of the imaging plane or pixel plane of the camera; preferably, the calibration plate is slidably mounted on one end of the slide rail, and the structured light module is fixedly mounted on the other end of the slide rail. Figure 2 As shown, the calibration plate is installed on the right side of the photosensitive element and the calibration plate is supported to slide left and right on the slide rail. The calibration plate is installed at the right end of the slide rail, and the structured light module is installed at the left end of the slide rail. Specifically, the structured light emitter and the camera are both installed at the left end of the slide rail, and the relative position between the structured light emitter and the camera is fixed by the fixture. Figure 2 The line between the center A of the structured light emitter and the optical center B of the camera is set as the baseline AB and is fixed, at least in the process of calibrating the parameters of the camera and the structured light; the slide rail does not have to be placed on a horizontal plane, but can also be placed on a vertical plane and can be rotated at will. Every time the calibration plate slides a certain distance along the slide rail, the light bar on the calibration plate is translated once along the vertical direction of the slide rail; combined with Figure 2It can be seen that the calibration plate starts to slide to the right from a starting position of the slide rail (set at a track position at a known distance from the camera). When the calibration plate slides to the position where the structured light emitted by the structured light emitter (from a two-dimensional plane perspective, generally from the perspective of the normal plane of the measured surface of the calibration plate, it is regarded as emitting a single-point line laser) intersects the measured surface of the calibration plate at point O, the reflected light at position point O is imaged to position E on the surface of the camera's photosensitive element through the optical center B of the camera's lens, so that the camera obtains the pixel coordinates at position E in real time, which are expressed as the center coordinates of the light bar projected in the pixel plane after the light bar is reflected from position point O; at the same time, the vertical distance CO from point O to the baseline AB is also determined, which is expressed as a test distance, which is also the distance between the calibration plate at the current position and the baseline AB, and can be expressed by the displacement of the calibration plate relative to the structured light module along the slide rail. At this time, the angle between the structured light emitted by the structured light emitter (from the perspective of a two-dimensional plane, generally from the perspective of the normal plane of the measured surface of the calibration plate, it is regarded as emitting a single-point line laser) and the baseline AB is ɑ, and the baseline AB is the line connecting the optical center B of the camera and the center A of the structured light emitter. When the calibration plate is set on the normal plane of the slide rail, the baseline AB is parallel to the measured surface of the calibration plate (regarded as perpendicular to the slide rail); then, when the angle between the structured light emitted by the structured light emitter (from the perspective of a two-dimensional plane, generally from the perspective of the normal plane of the measured surface of the calibration plate, it is regarded as emitting a single-point line laser) and the baseline AB is fixed at ɑ, the calibration plate is only driven to slide away from the camera until the structured light emitted by the structured light emitter (from the perspective of a two-dimensional plane, generally from the perspective of the normal plane of the measured surface of the calibration plate, it is regarded as emitting When a single-point line laser (laser) intersects the measured surface of the calibration plate at the position of point O', the distance between position point O' and position point O is equal to y, and the reflected light at position point O' is imaged to position F on the surface of the camera's photosensitive element through the optical center B of the camera lens, that is, in the process of the calibration plate deviating from the fixed position of the camera along the slide rail from near to far, the imaging position of the reflected light on the surface of the camera's photosensitive element is shifted upward from position E along the longitudinal axis to position F, and the camera obtains the pixel coordinates at position F in real time, which is expressed as the center coordinates of the light bar projected in the pixel plane after the light bar passes through the position point O' of the calibration plate; at the same time, the vertical distance C'O' from point O' to the baseline AB is also determined, which is expressed as a test distance, which is also the distance between the calibration plate at the current position and the baseline AB, and can be expressed by the displacement of the calibration plate relative to the structured light module along the slide rail. On this basis, the distance CO and the pixel coordinates at position E can be used as a vector group to participate in the nonlinear fitting of step 2, and the distance C'O' and the pixel coordinates at position F can also be used as a vector group to participate in the nonlinear fitting of step 2, which is used to fit the relevant modeling parameters.Therefore, the calibration plate calibrates the structured light module under the drive of the motor. There is no need to specially set up targets such as a checkerboard. The calibration can be performed by simply controlling the calibration plate to slide along the slide rail, which is easy to automate. This improves the efficiency of the structured light module calibration while ensuring the reliability of the calibration results of the structured light module, that is, while ensuring the accuracy of the calibration parameters of the structured light module.

[0039] In this embodiment, the calibration plate, the slide rail, and the fixture for fixing the sensor in the structured light module can form a calibration device for calibrating the structured light module under the drive of a motor, wherein the structured light module can collect two-dimensional images of the calibration plate at different distances, and the light strips in a certain frame of the two-dimensional image will be clearly imaged in the imaging plane of the camera and will not deviate from the imaging plane of the camera (the surface of the photosensitive element). When the structured light module changes, including at least one of the focal length of the camera, the relative position between the structured light emitter and the camera, the emission direction of the structured light emitter (whose change will affect whether a light strip can be projected on the calibration plate at a certain distance), and the optical axis direction of the camera, the calibration plate needs to be driven by a motor to slide to the corresponding position before the structured light emitted by the structured light emitter can be incident on the calibration plate and the reflected light can be imaged through the camera lens to the surface of the camera's photosensitive element. Only then can the camera capture the image containing the light strip and then calibrate the camera. Therefore, it is necessary to constrain the sliding distance of the calibration plate on the slide rail, such as constraining the calibration plate to slide only one step distance each time under the drive of the motor to meet the camera imaging requirements.

[0040] It should be noted that the size of the calibration plate and the length of the slide rail should be adaptively designed according to the relative position between the camera and the structured light emitter and the ranging range limited on the slide rail to ensure that the posture information within the effective measurement range of the structured light module is covered.

[0041] Preferably, when the camera is a distortion-free camera, when the surface of the camera's photosensitive element and the measured surface of the calibration plate form a certain oblique angle, the distance between the baseline and the calibration plate can be directly calculated according to the pinhole imaging principle. The surface of the camera's photosensitive element and the measured surface of the calibration plate can be regarded as parallel. In this way, the distance between the structured light module and the calibration plate can be directly calculated according to the pinhole imaging principle, which will not affect the calibration accuracy of the camera.

[0042] As an embodiment, in the step 1, during the process of the calibration plate sliding along the slide rail from the upper limit of the range to the lower limit of the range, or during the process of the calibration plate sliding along the slide rail from the lower limit of the range to the upper limit of the range, each time the calibration plate slides a preset fixed distance, the camera captures the image of the calibration plate, and the imaging position of the light bar in the imaging plane is offset by a preset height along the height direction. Specifically, the imaging position of the light bar at a corresponding test angle is offset by a preset height along the height direction, that is, relative to the imaging position in the image captured before the current sliding, it is offset by a preset height along the height direction, and the corresponding projected center coordinate of the light bar is offset once in the opposite direction of the height direction in the pixel plane. It can be expressed by the number of pixels, and sub-pixel level is also allowed, indicating that the offset on the pixel is not necessarily an integer, but can also be a decimal, and is detected by the camera in real time.

[0043] refer to Figure 2 It can be known that when the calibration plate slides until the structured light emitted by the structured light emitter (from the perspective of a two-dimensional plane, generally from the perspective of the normal plane of the measured surface of the calibration plate, it is regarded as emitting a single-point line laser) intersects the calibration plate at position point O, and also when the intersection line of the structured light surface formed by the structured light emitter and the measured surface of the calibration plate passes through position point O, the offset formed by the imaging position E of the light bar in the imaging plane relative to the reference position D is the light spot offset DE, represented by x in the figure, as the light spot offset corresponding to a calibration column or a test angle; wherein, during the sliding of the calibration plate on the slide rail, the line between the center of the structured light emitter and the center of the camera remains parallel to the calibration plate. The motor driving the calibration plate to slide is a stepper motor. At each position where the calibration plate slides driven by the stepper motor, the distance of each sliding of the calibration plate is the stepping distance generated by the stepper motor control, and the stepping distance is a preset fixed distance. When the calibration plate steps the same distance along the slide rail, the distance of the calibration plate relative to the structured light module and its change are both measurable and recorded. The offset of the light bar on the calibration plate is equal and can be within the visible range of the camera, which is also equivalent to the detectable area of ​​the structured light module. The light spot offset and its change are both detected in real time, and the center coordinates of the light bar are obtained, so as to obtain multiple sets of vectors for nonlinear fitting in step 2, each set of vectors is the center coordinates of the light bar of a calibration column of a frame of calibration plate image and its corresponding test distance, and multiple sets of vectors are the center coordinates of the light bar of the same calibration column of all frames of calibration plate images and their corresponding test distances, so as to improve the accuracy of the modeling parameters corresponding to the same calibration column fitted.

[0044] Specifically, when the calibration plate approaches the structured light module, that is, when it approaches the structured light module in steps according to a preset fixed distance, the calibration plate slides along the slide rail from the upper limit of the range to the lower limit of the range with a preset fixed distance as the stepping distance. The imaging position of the light bar in the imaging plane is shifted in the height direction with a preset height as the stepping distance, which can be detected by the camera in real time; the center coordinates of the light bar formed by the projection of the light bar in the pixel plane are shifted in the opposite direction of the height direction with a preset longitudinal offset as the stepping distance. The reason is that the changes in the pixel plane and the changes on the imaging surface are reversed. For example, the imaging position of the light bar in the imaging plane is shifted upward once along the longitudinal axis, and the center coordinates of the light bar formed by the projection of the light bar in the pixel plane are shifted downward once along the longitudinal axis; the camera detects the center coordinates of the light bar in the pixel coordinate system or pixel plane, preferably starting from a pixel offset starting point in the pixel plane of the camera, and measured with a preset longitudinal offset as the stepping distance.

[0045] Specifically, when the calibration plate is away from the structured light module, that is, when it moves away from the structured light module in steps according to a preset fixed distance, the calibration plate slides along the slide rail from the lower limit of the range to the upper limit of the range with a preset fixed distance as the stepping distance, and the imaging position of the light bar in the imaging plane is offset in the height direction with a preset height as the stepping distance, which can be detected by the camera in real time; the center coordinates of the light bar formed by the projection of the light bar in the pixel plane are offset in the opposite direction of the height direction with a preset longitudinal offset as the stepping distance, because the changes in the pixel plane and the changes on the imaging surface are reversed; for example, if the imaging position of the light bar in the imaging plane is offset upward once along the longitudinal axis, then the center coordinates of the light bar formed by the projection of the light bar in the pixel plane are offset downward once along the longitudinal axis; the camera detects the center coordinates of the light bar in the pixel coordinate system or pixel plane, preferably starting from a pixel offset starting point in the pixel plane of the camera, and measured with a preset longitudinal offset as the stepping distance.

[0046] In some embodiments, on the slide rail, the initial position of the calibration plate is placed at the lower limit of the range, that is, the structured light module is fixedly mounted at one end of the slide rail, and the calibration plate is slidably mounted on the other end of the slide rail, and is driven by a stepper motor to step toward the upper limit of the range of the slide rail in sequence. Here, the constraint on the range measurement range defined on the slide rail can be determined based on whether the light bar is clearly imaged on the photosensitive element of the camera, at least to ensure that the image coordinate position (the coordinate position in the image coordinate system (i.e., the imaging plane coordinate system)) projected by the light bar does not deviate from the surface area of ​​the photosensitive element, such as the farthest position from the camera that the calibration plate is allowed to slide on the slide rail is the farthest position that supports the light bar projected on the calibration plate by the structured light emitted by the structured light emitter to focus on the camera. When the calibration plate is at the upper limit of the range of the slide rail, the imaging position of the light bar in the imaging plane is set as the reference position, wherein the upper limit of the range of the slide rail is the farthest position from the structured light module (which can also be regarded as the camera) that the calibration plate is allowed to slide on the slide rail.

[0047] In other embodiments, the reference position is the imaging position of the reflected light on the measured surface of the measured object farthest from the structured light module (which can also be regarded as the camera) emitted by the structured light emitter in the detectable area of ​​the structured light module, which is recorded as the limit position of the reflected light imaging on the surface of the photosensitive element when the measured surface of the measured object is infinitely far from the baseline; the angle ɑ formed by the structured light emitted by the structured light emitter and the baseline AB affects the setting of the reference position; preferably, corresponding to the same calibration column, the single-point line laser emitted by the structured light emitter is parallel to the line connecting the reference position and the optical center of the camera; when the center A of the structured light emitter remains unchanged and the optical center B of the camera remains unchanged, the angle ɑ increases, that is, the ray AO' rotates counterclockwise around A, and the reference position D shifts downward along the longitudinal axis; when the center A of the structured light emitter remains unchanged and the optical center B of the camera remains unchanged, the angle ɑ decreases, that is, the ray AO' rotates clockwise around A, and the reference position D shifts upward along the longitudinal axis. Wherein, the measured object includes the calibration plate. In summary, the reference position is used to represent the offset reference point of the projection position of the light strip in the imaging plane coordinate system, providing an imaging position offset starting point for the measured object or calibration plate at any distance, which is convenient for measuring and describing the displacement of the light strip in the imaging plane or in the photosensitive element from the reference position, thereby improving the calibration efficiency.

[0048] As an embodiment, the upper limit of the range of the slide rail is the position farthest from the structured light module (which can also be regarded as the camera) that the calibration plate is allowed to slide on the slide rail, and its distance relative to the structured light module or the baseline is obtained in advance, and is within the detectable area of ​​the structured light module. The lower limit of the range of the slide rail is the position closest to the structured light module (which can also be regarded as the camera) that the calibration plate is allowed to slide on the slide rail, and its distance relative to the structured light module or the baseline is obtained in advance, and is within the detectable area of ​​the structured light module. The calibration plate slides along the slide rail between the upper limit and the lower limit of the range of the slide rail, and the structured light emitted by the structured light emitter is incident on the measured surface of the calibration plate, and the reflected light returns to the optical path formed by the lens of the camera along a test angle, which is within the detectable area of ​​the structured light module. On the slide rail, when the initial position of the calibration plate is placed at the upper limit or the lower limit of the range, when the calibration plate slides to a position on the slide rail, the offset formed by the imaging position of the light strip in the imaging plane relative to the reference position is marked as the light spot offset, specifically, the displacement of the pixel point in a calibration column from the reference position in the imaging plane coordinate system. Since the calibration plate will slide from the initial position to a position, the distance it slides and the position reached by the calibration plate are obtained in real time to determine the distance between the baseline and the calibration plate; since the imaging position of the light strip in the imaging plane is The preset height is set as the step distance and the light bar is offset along the height direction, so the imaging position of the light bar in the imaging plane is obtained in real time, specifically, it is calculated and determined from the coordinate information of the imaging position of the light bar reflected from the calibration plate at the initial position in the imaging plane; since the light bar center coordinates formed by the projection of the light bar in the pixel plane are offset in the opposite direction of the height direction with the preset longitudinal offset as the step distance, the light bar center coordinates formed by the projection of the light bar in the pixel plane are obtained in real time, specifically, it is calculated and determined from the coordinate information of the pixel position projected from the reflected light of the light bar at the initial position of the calibration plate in the pixel plane. Wherein, when the relative position of the structured light emitter and the camera remains unchanged, when the calibration plate is displaced relative to the baseline, the light spot offset changes, and the light bar center coordinates change. Thus, the sliding range of the calibration plate on the slide rail is constrained by the upper limit of the range of the slide rail and the lower limit of the range of the slide rail to limit the test distance, and then the offset range of the imaging position of the light bar in the imaging plane is limited on the calibration plate, so as to obtain the light bar center coordinates and corresponding test distance that support nonlinear fitting within a reasonable range.

[0049] As an embodiment, under the same calibration column, the method of constructing a structured light ranging relationship based on the pinhole imaging principle includes: firstly, the mathematical model between the center coordinates of the light strip to be established and the test distance (i.e., the aforementioned structured light ranging relationship) is explained, which is also used as the function to be fitted in the nonlinear fitting. For ease of understanding, the explanation starts with the principle of single-point laser triangulation ranging. From the perspective of a two-dimensional plane, generally the perspective of the normal plane of the measured surface of the calibration plate, the structured light emitted by the structured light emitter is Figure 2 In the figure, it is considered as emitting a single-point line laser, which is equivalent to the single-point line laser light path diagram. When the structured light emitted by the structured light emitter in step 1 is incident on the calibration plate at a certain angle to the normal of the measured surface of the calibration plate, the reflected light is imaged into the photosensitive element of the camera through the lens at position B. B is the optical center of the camera and also the pinhole in the pinhole model.

[0050] like Figure 2 As shown, in this embodiment, the line between the center A of the structured light emitter and the optical center B of the camera is set as the baseline, and the length of the baseline is set as the baseline distance AB; the baseline distance AB is predetermined in this embodiment, and is an assembly parameter of the structured light module, and is also an assembly parameter in the height direction. In some embodiments, when the distance between the structured light module and the calibration plate is represented by the horizontal distance between the line between the center of the structured light emitter and the optical center of the camera and the measured surface of the calibration plate, it can represent the displacement of the calibration plate relative to the structured light module along the slide rail when the slide rail is placed horizontally, and can also represent the horizontal component of the displacement of the calibration plate relative to the structured light module along the slide rail when the slide rail is placed tilted to the horizontal plane.

[0051] In this embodiment, the distance between the baseline and the calibration plate is set as the test distance. For example, when the calibration plate slides to the position where the structured light emitted by the structured light emitter (from the perspective of a two-dimensional plane, generally from the perspective of the normal plane of the measured surface of the calibration plate, it is regarded as emitting a single point line laser) intersects the measured surface of the calibration plate at point O, the reflected light at position point O is imaged to position E on the surface of the camera's photosensitive element through the optical center B of the camera's lens, so that the camera obtains the pixel coordinates at position E in real time, which are expressed as the center coordinates of the light bar projected in the pixel plane after the light bar is reflected from position point O. Figure 2 The distance OC represents the test distance.

[0052] The structured light ranging relationship constructed according to the pinhole imaging principle is: Px=k0 / d+k1, which is actually the relationship between the center coordinates of the light bar and the test distance under the same calibration column or the same test angle, where d is the test distance, Px is the center coordinates of the light bar, and the analysis here is based on the X-axis component (associated with the baseline distance AB), the unit is pixel, and it is expressed as the number of pixels. The coefficients required to construct the structured light ranging relationship are k0 and k1, respectively, which are used as the modeling parameters. Specifically, during the sliding of the calibration plate on the slide rail, the center coordinates Px of the light bar, the baseline distance AB, the focal length f of the camera, and the test distance OC are all measured or calculated in advance, and the emission angle of the structured light emitter and the optical axis direction of the camera are predetermined. The reflected light path of the structured light is determined from the perspective of the two-dimensional plane, and the structured light ranging relationship is constructed using the pinhole imaging principle without converting the three-dimensional coordinate system to the two-dimensional coordinate system.

[0053] Under the same calibration column or the same test angle, based on the pinhole imaging principle, the ratio of the baseline distance to the test distance is equal to the ratio of the light spot offset to the focal length of the camera. Figure 2 It can be seen that the position D on the surface of the camera's photosensitive element is used as the reference position, where the line connecting the reference position D and the camera's optical center B is parallel to the structured light emitted by the structured light emitter at position A (from a two-dimensional plane perspective, generally from the perspective of the normal plane of the measured surface of the calibration plate, it is regarded as emitting a single point line laser), and the structured light emitted by the structured light emitter is parallel to the baseline AB (also regarded as the baseline distance AB). Figure 2 It can be seen that when the calibration plate slides until the structured light emitted by the structured light emitter (from the perspective of the two-dimensional plane, generally from the perspective of the normal plane of the measured surface of the calibration plate, it is regarded as emitting a single point line laser) intersects with the calibration plate at position point O, and when the intersection line of the structured light surface formed by the structured light emitter and the measured surface of the calibration plate passes through position point O, the offset formed by the imaging position E of the light bar in the imaging plane relative to the reference position D is the light spot offset DE, which is represented by x in the figure; the distance between the baseline AB and the calibration plate, that is, the test distance is represented by OC. Since the baseline AB is parallel to the calibration plate, the surface of the camera's photosensitive element is regarded as parallel to the baseline, so based on the pinhole imaging principle, the triangle ABO and the triangle DEB are similar, so there is: AB / OC=x / f. Therefore, x=(AB*f) / OC, where OC supports real-time measurement during the sliding of the calibration plate on the slide rail. Thus, a calculation model for the light spot offset is established through the baseline distance, focal length and test distance.

[0054] The light spot offset x is equal to the product of the center coordinate Px of the light strip and the single pixel size CellSize and the sum of the projection error Dev, that is, x=Px*CellSize+Dev. The premise for this relationship is that the image of the light strip in the imaging plane of the camera is set in the imaging plane coordinate system, and the reference position and the light spot offset are both set in the imaging plane coordinate system. The imaging plane coordinate system here can also be understood as the image coordinate system of the camera coordinate system; the center coordinate of the light strip is set in the pixel coordinate system, and the pixel coordinate system is the coordinate system of the pixel plane of the camera. The projection error Dev represents the difference between the light spot offset acquired by the camera in real time and the light spot offset formed by the actual projection in the imaging plane of the camera for a light strip on the surface of the object being measured, which is the result obtained in advance. Figure 2 In the equation, combined with x=(AB*f) / OC, the obtained structured light ranging relationship is Px=[(AB*f / OC)-Dev] / CellSize; then, k0 is represented by (AB*f) / CellSize; at the same time, k0 is represented by -Dev / CellSize; at this point, the structured light ranging relationship can be simplified to Px=k0 / d+k1. In this embodiment, k0 and k1 are both modeling parameters to be determined. Therefore, the camera intrinsic parameters that originally need to be calibrated are simplified from Dev, CellSize, AB and f to k0 and k1, which not only simplifies the ranging relationship that requires coordinate system conversion under the pinhole imaging principle, but also reduces the parameters that need to be calibrated. The functional relationship between the center coordinates of the light strip in the photosensitive element and the test distance (the distance between the measured surface (target surface) of the calibration plate and the baseline) is simplified.

[0055] In summary, the size of a single pixel and the projection error are both unknown quantities, so k0 and k1 both constitute unknown quantities in the structured light ranging equation and need to be fitted. Figure 2The triangular ranging relationship constructed by a pair of similar triangles, triangle ABO and triangle DEB, is simplified, especially in the case of converting from the imaging plane coordinate system to the pixel coordinate system, reducing the types and number of parameters that need to be calculated, which is equivalent to reducing the number of camera internal parameters that need to be calibrated, simplifying the camera calibration method. Moreover, when the calibration plate and the baseline produce relative displacement, x will change accordingly, and then Px will produce displacement. Therefore, it is only necessary to determine the two parameters k0 and k1 in this structured light ranging relationship to complete the modeling of the ranging model of the structured light module. Therefore, this embodiment only needs to determine the two parameters k0 and k1 in the aforementioned structured light ranging relationship to complete the modeling of the structured light module or fit the corresponding function relationship, that is, to establish a model for measuring the distance between the baseline and the object being measured using structured light, which is equivalent to completing the calibration of the camera and simplifying the camera calibration method. Then, on the basis of simplifying the camera intrinsic calibration operation, the functional relationship d=k0 / (Px-k1) between the test distance and the center coordinates of the light strip is determined, which is used to calculate the test distance when the center coordinates of the light strip are known as the ranging information of the structured light module.

[0056] This application needs to obtain the structured light ranging relationship, so it is necessary to understand how the camera itself obtains the world coordinates in the three-dimensional space step by step from the pixel coordinates. Here, the following coordinate systems are involved: World coordinate system: A reference coordinate system is also selected in the environment to describe the position of the camera and the object being measured. This coordinate system is called the world coordinate system. The relationship between the camera coordinate system and the world coordinate system can be described by a rotation matrix and a translation vector. In this application, the coordinate system where the structured light module or structured light emitter is located can be used as the world coordinate system.

[0057] Camera coordinate system: The origin of the camera coordinate system is the optical center of the camera. The x-axis and y-axis are parallel to the X and Y axes of the image. The z-axis is the optical axis of the camera, which is perpendicular to the image plane. The spatial rectangular coordinate system formed by this is called the camera coordinate system, also known as the camera coordinate system. The camera coordinate system is a three-dimensional coordinate system. The intersection of the optical axis and the image plane is the origin of the image coordinate system. The rectangular coordinate system formed by the X and Y axes of the image is the image coordinate system. The image coordinate system is a two-dimensional coordinate system. If the world coordinates are converted to camera coordinates, there is a rigid body transformation. In fact, it corresponds to the camera extrinsic parameters. The camera coordinates are converted to image coordinates, which is a 3D->2D perspective projection process.

[0058] The imaging plane coordinate system (X, Y) is the image coordinate system XOY in the aforementioned camera coordinate system. The units of its coordinate axes are usually millimeters (mm). The origin is the intersection of the camera optical axis and the image plane (called the principal point), that is, the center point of the image. The X-axis and Y-axis are parallel to the U-axis and V-axis respectively.

[0059] The pixel coordinate system (U, V) is a two-dimensional rectangular coordinate system that reflects the arrangement of pixels on a two-dimensional image plane. The U axis and the V axis are parallel to the two sides of the image plane. The unit of the coordinate axis in the pixel coordinate system is pixel (integer). The image coordinates are converted into pixel coordinates, and the image coordinate system is discretized. The origin of the pixel coordinate system is generally located in the upper left corner of the image. The offset from the optical center to the origin of the pixel coordinate system is recorded as the reference pixel coordinate, that is, the coordinate of the camera optical center in the pixel coordinate system. The size of each pixel in the horizontal and vertical directions is recorded as (mm / pixel).

[0060] As an embodiment, under the control of the controller of the aforementioned structured light module, the motor drives the calibration plate to slide on the slide rail, and in the process of the calibration plate sliding along the slide rail from the upper limit of the range to the lower limit of the range, it slides through multiple positions in sequence at least according to a preset fixed distance (the step distance driven by the motor), preferably 8 positions, and all are located between the upper limit of the range of the slide rail and the lower limit of the range of the slide rail; at each position, the camera captures a frame of the calibration plate image and extracts the center coordinates of the light bar in the corresponding calibration column, including the center coordinates of the light bar corresponding to a pixel point in the pixel coordinate system; when the relative position of the structured light emitter and the camera remains unchanged, when the calibration plate is displaced relative to the baseline, the light spot offset changes, and the center coordinates of the light bar change. Therefore, in step 2, the method for nonlinear fitting of the center coordinates of the light bar of the same calibration column and the test distance corresponding to the same calibration column includes: Step 21, sequentially traverse the i-th calibration column of the M frames of calibration plate images acquired in step 1; generally, start from the first column of pixels of the first frame of calibration plate image acquired by the camera at the initial position of the slide rail. Then execute step 22.

[0061] Step 22, extract the row pixel position coordinate Pxj of the center of the light bar of the ith calibration column from the currently traversed frame of the calibration plate image, and record it as the center coordinate of the light bar of the ith column of the currently traversed frame of the calibration plate image. The row pixel position coordinate of the center of the light bar of the ith calibration column can be understood as the coordinate position of the center of gravity of the light bar of the ith calibration column in the pixel plane, which can be extracted in real time or calculated by a related algorithm; at the same time, determine the test distance dj corresponding to the ith calibration column of the same frame of the calibration plate image, which can be measured from the slide rail; j represents the sorting number of the currently traversed frame of the calibration plate image, so that the currently traversed frame of the calibration plate image is marked as the jth frame of the calibration plate image; j is a positive integer, j is greater than 0, j is less than or equal to M, M is also a positive integer, preferably 8 or a greater value. Then execute step 23.

[0062] Step 23, configure j=j+1, update the data of the j+1th calibration plate image frame to the data of the jth calibration plate image frame, and repeat step 22 until the i-th calibration column of the M-frame calibration plate image is traversed, and the center coordinates of the i-th column of the light bar in each frame of the calibration plate image are combined into a coordinate set {Px1, Px2,…, PxM}, and at the same time, the test distance corresponding to the i-th calibration column in each frame of the calibration plate image is combined into a distance set {d1, d2,…, dM} or a distance set {1 / d1, 1 / d2,…, 1 / dM} to adapt to the aforementioned relationship Px=k0 / d+k1, and then execute step 24.

[0063] In step 23, k0 and k1 are both unknown quantities, Px and d are both known quantities, Px includes Px1, Px2, ..., PxM; d includes d1, d2, ..., dM. Among them, j in step 21 is incremented from the value 1, so the configuration of step 23 j=j+1 is equivalent to traversing the next frame of the calibration plate image in step 21.

[0064] Preferably, the center coordinates of any two light bars in {Px1, Px2, ..., PxM} are not equal, and any two test distances in {d1, d2, ..., dM} are not equal.

[0065] Step 24, configure the coordinate set of step 23 and the distance set of step 23 as data sets on two coordinate axes (i.e., two sets of input data sets of the function to be fitted), perform nonlinear fitting, obtain the modeling parameters k0 and k1 corresponding to the ith calibration column, determine that the calibration of the modeling parameters corresponding to the ith calibration column is completed; and determine to fit the structured light ranging relationship corresponding to the ith calibration column. Whenever the structured light ranging relationship corresponding to the ith calibration column is fitted, it is determined to find the relationship between the center coordinate Px of the light bar and the test distance d, and it is not necessary to calibrate the camera internal parameters in advance, or it is understood to simplify the camera calibration process; and record the modeling parameters k0 and k1 corresponding to the ith calibration column as a modeling parameter pair of the ith calibration column, i.e., the modeling parameters corresponding to the ith calibration column. At this time, in the aforementioned relationship Px=k0 / d+k1, k0 and k1 become known quantities, while Px and d become unknown quantities. Then execute step 3.

[0066] The modeling parameters k0 and k1 corresponding to the same calibration column finally fitted in step 24 satisfy the structured light ranging relationship corresponding to the same calibration column collected by the calibration plate moving along the slide rail, and are further extended to meet the ranging requirements of the structured light module for the object under test at the corresponding test angle, that is, the fitting coefficients required for the ranging model of the structured light module at the corresponding test angle.

[0067] It should be noted that the modeling parameters fitted by the modeling parameter calibration method are necessary parameters in the structured light ranging relationship followed by the same calibration column of each frame of the calibration plate image and the corresponding light bar center coordinates during the sliding process of the calibration plate. Therefore, in order to construct this structured light ranging relationship on the same calibration column of each frame of the calibration plate image, it is necessary to introduce the light bar center coordinates projected at multiple distance positions for fitting, so as to realize the calibration of the corresponding modeling parameters in the same calibration column of each frame of the calibration plate image.

[0068] In summary, this embodiment does not need to convert into an intrinsic parameter matrix according to the intrinsic parameter calibration method disclosed in the prior art, and does not need to obtain the actual space coordinates through the light plane parameters and the camera intrinsic parameters like the traditional method, but relies on the motor to drive the calibration plate to slide to different distances to collect the center coordinates of the light strips under the same calibration column and their corresponding test distances, and fit the two corresponding modeling parameters under the same calibration column. Moreover, these are two modeling parameters fitted by multiple frames of calibration plate images under the same calibration column, rather than multiple optical parameters corresponding to multiple frames of calibration plate images under the same calibration column. Compared with the prior art, the types and quantities of camera intrinsic parameters that need to be calculated are reduced, thereby simplifying the calculation amount of the transformation relationship from the camera coordinate system to the pixel coordinate system, and improving the calibration efficiency of the structured light module; when it is subsequently actually applied to ranging, the calculation efficiency is also improved.

[0069] On the basis of the above embodiment, every time the calibration of the modeling parameters corresponding to a calibration column is completed by executing step 2, step 3 is executed once. In step 3, i=i+1 is configured so that the pixels of the i+1th calibration column in the jth frame calibration plate image are updated to the pixels of the ith calibration column in the jth frame calibration plate image, and then steps 21 to 24 are repeated until the N calibration columns in each frame calibration plate image collected in step 1 are traversed, and it is determined that after all calibration columns are nonlinearly fitted, N modeling parameter pairs are accumulated to determine that the calibration of the modeling parameters of the structured light module is completed; wherein, i in step 21 is incremented from the value 1, therefore, step 3 configuration i=i+1 is equivalent to step 21 traversing the next calibration column of the same frame calibration plate image. Specifically, in M ​​frames of calibration plate images, by repeatedly executing steps 21 to 24, N structured light ranging relational expressions are cumulatively fitted; N modeling parameter pairs correspond to N calibration columns in each frame of calibration plate images, and the N calibration columns are N columns calibrated in the pixel plane of the camera in the same frame of calibration plate images, and each column corresponds to a test angle and a test distance; i represents the column number of a calibration column currently traversed in a frame of calibration plate images, i is a positive integer, i is greater than 0, and i is less than or equal to N. Therefore, there are N columns of pixel points or N calibration columns of light bar center coordinates in the camera image space, and the methods of the aforementioned steps 2 and 3 can be used to perform simple modeling at N test angles in turn, and N structured light ranging relational expressions and modeling parameter pairs are cumulatively fitted, that is, 2*N modeling parameters are cumulatively fitted, and the camera calibration is completed. Therefore, this embodiment repeatedly executes step 21 to step 24 to complete the calibration of the corresponding modeling parameters column by column in each frame of the calibration plate image, thereby completing the calibration of the structured light module from multiple test angles.

[0070] Preferably, the nonlinear fitting adopts the least squares method; specifically, the test distance corresponding to the center coordinates of the i-th column of the j-th frame calibration plate image and the i-th calibration column of the j-th frame calibration plate image supports fitting processing using the least squares method, and the mapping relationship corresponding to the i-th calibration column is fitted, so that the optimal coefficient is fitted through the least squares method as the aforementioned modeling parameter, so that the fitting error is minimized.

[0071] The present application also discloses a chip, which is used to execute the modeling parameter calibration method. As the execution subject of steps 1, 2 and 3 in the above-mentioned embodiment, the chip can not only drive the calibration plate on the slide rail to displace relative to the structured light module by controlling the motor, but also control the light strips formed by the structured light emitted by the structured light module on the surface of the object to be measured, and also control the camera including the image of the light strips, so that the coordinates of the pixel position projected on the pixel plane of the structured light module are detected by the camera in real time, then the chip is equivalent to a common controller of the structured light module and the motor. Furthermore, the chip can also perform steps 1 to 3 on the ranging system composed of the single-point line laser emitter and the camera at each test angle in turn to perform modeling on each calibration column, that is, at each test angle, the modeling parameters on the corresponding calibration column are obtained by nonlinear fitting, avoiding the calibration of the camera internal parameters first.

[0072] In this embodiment, during the process of performing nonlinear fitting on the center coordinates of the light stripes of the same calibration column and the test distance corresponding to the same calibration column, the optimal relationship between the center coordinates of the light stripes and the test distance is fitted and the modeling parameters are obtained, wherein the center coordinates of the light stripes and the test distance (the distance of the calibration plate relative to the structured light module) extracted from the calibration plate image at different distances are known quantities that can be acquired in real time from each frame of the calibration plate image. Therefore, these known quantities can be used to fit and calculate the coefficients required to construct the structured light ranging relationship from the image data of multiple frames of the calibration plate image, thereby establishing a mathematical fitting model for the structured light module ranging.

[0073] Therefore, the chip can calculate the aforementioned modeling parameters by performing nonlinear fitting on the mapping relationship between the center coordinates of the light stripes and their test distances in the same calibration column of multiple frames of calibration plate images, as the optimal parameters for constructing the mapping relationship between the center coordinates of the light stripes and their test distances in the same calibration column, that is, the coefficients required to construct the structured light ranging relationship. The fitted modeling parameters can replace the traditional camera intrinsic parameters in a direction or a test angle relative to the optical axis of the camera, so as to replace or omit the calibration of the camera intrinsic parameters before the ranging operation.

[0074] In summary, the aforementioned chip performs steps 1 to 3, controls the displacement of the calibration plate relative to the structured light module, and uses the center coordinates of the light bars of different calibration columns in the calibration plate images collected at different distances and their corresponding test distances for nonlinear fitting to complete parameter calibration. The fitted modeling parameters can replace the traditional camera internal parameters in a direction or a test angle relative to the optical axis of the camera, thereby improving the efficiency of the calibration of the structured light module while ensuring the accuracy of the calibration parameters of the structured light module. In addition, the modeling parameters corresponding to each calibration column fitted by the chip and the structured light ranging relationship that can be constructed, the ranging modeling method between the structured light and the camera or the calibration method of the camera are simplified to only fitting and calculating the modeling parameters corresponding to each calibration column, and then replace or omit the calibration of the camera internal parameters before the ranging operation, so that the camera calibration calculation amount is less, and the calibration efficiency of the structured light module is improved; further, after obtaining the modeling parameters, the modeling and ranging operations can be performed without calibrating the camera internal parameters, and the calculation efficiency is improved.

[0075] The present application also discloses a calibration device, which includes a calibration plate, a slide rail and the chip disclosed in the aforementioned embodiment, and the calibration device is used to calibrate the structured light module. In the calibration device, the chip is used to control the calibration plate to slide along the slide rail, specifically, the calibration plate is driven by a motor to slide within a predetermined range, and the structured light emitter is controlled to emit structured light to the measured surface of the calibration plate, and the camera is controlled to collect the image of the structured light projected onto the measured surface of the calibration plate, and a frame of image including light strips is collected at different distances from the structured light module, that is, the aforementioned calibration plate image, so as to obtain the modeling parameters corresponding to each calibration column in each frame of the calibration plate image by executing steps 1 to 3, and determine that the calibration of the modeling parameters of the structured light module is completed.

[0076] Among them, the structured light module includes a structured light emitter and a camera. Specifically, the structured light emitted by the structured light emitter can be composed of multiple single-point line lasers, each single-point line laser corresponds to a calibration column and also corresponds to a test angle formed relative to the optical axis of the camera.

[0077] The modeling parameters corresponding to the same calibration column fitted by the calibration device disclosed in the present application are the coefficients required to construct the structured light ranging relationship under the same calibration column. The structured light ranging relationship is the relationship between the center coordinates of the light strips in the same calibration column and the corresponding test distances constructed based on the pinhole imaging principle. Constructing / defining the structured light ranging relationship under the pinhole imaging principle can be specifically understood as the relationship between the center coordinates of the light strips constructed under the pinhole imaging principle and the corresponding test distances. The coefficients required to construct the structured light ranging relationship can be fitted and calculated in combination with the data of the calibration plate image at different distances. In this embodiment, the structured light ranging relationship can be used as the corresponding function to be fitted in the nonlinear fitting to fit the modeling parameters, including at least two fitting coefficients. Therefore, compared with the line laser-camera modeling ranging scene in the prior art, this embodiment converts the mathematical formula constructed by the triangle ranging into the relationship between the center coordinates of the light strips and the corresponding test distances, without the need to additionally calibrate the internal and external parameters of the camera, and the calibration calculation amount is less.

[0078] When the Zhang calibration method disclosed in the prior art is used for camera calibration, a calibration board with a chessboard layout is used for camera calibration. The detection error of the chessboard corner points is high, resulting in a large error in the subsequent internal parameter matrix. The rotation matrix in the external parameter no longer satisfies the constraints of the special orthogonal group, and the internal and external parameter matrices obtained are inaccurate. Compared with the prior art, the calibration device disclosed in this application has low requirements on the structure to be built, and the camera calibration model used is simplified. It can be widely used in many scenarios such as rapid measurement of product dimensions and rapid guidance of robot trajectories in various fields such as automobile production and electronic manufacturing.

[0079] Preferably, the calibration device further comprises a fixture, which is used to fix the line between the center of the structured light emitter and the center of the camera to be parallel to the measured surface of the calibration plate, and is also used to assemble the emission direction of the structured light emitter and the optical axis direction of the camera to be set at an angle; the distance between the center of the structured light emitter and the center of the camera is fixed. The emission direction of the structured light emitter intersects with the calibration plate, so that the structured light emitted by the structured light emitter is incident on the calibration plate at a certain angle to the normal of the measured surface of the calibration plate, and the reflected light is imaged into the photosensitive element of the camera through the lens of the camera, wherein the camera collects the image of the reflected light of the structured light in the photosensitive element of the camera, and uses the image of the light bar in the imaging plane of the camera to represent it; the image of the light bar in the imaging plane of the camera is part of the image of the calibration plate; it is worth noting that the camera is configured to be free of distortion. Therefore, this embodiment reduces distortion calculation, and supports the use of the pinhole imaging principle to accelerate the establishment of the structured light ranging relationship, thereby accelerating calibration.

[0080] The relative position between the center of the structured light emitter and the center of the camera, and the angle between the emission direction of the structured light emitter and the optical axis direction of the camera are fed back to the chip or structured light module. The chip can fix the relative position and angle by controlling the fixture, thereby maintaining the relative posture between the camera and the structured light emitter unchanged.

[0081] It should be noted that the above description is only for the convenience of explaining the relative positions of the components in the structured light module. When the relative positions remain unchanged, the structured light module and the calibration plate do not necessarily have to be placed on the horizontal plane, but can also be placed on the vertical plane; driven by the motor, the calibration plate can move freely or rotate relative to the slide rail, and the placement posture is unconstrained.

[0082] When the surface of the camera's photosensitive element (photosensitive surface) is preferably parallel to the measured surface of the calibration plate, the object plane (from the calibration plate) and the image plane (camera's photosensitive element) can remain parallel, making it convenient to use the pinhole imaging principle to establish similar triangles for triangular ranging calculations, thereby speeding up calibration by using the triangulation ranging method commonly used in single-point line laser ranging.

[0083] The plane where the calibration plate image captured by the camera is located can be regarded as being decomposed into columns. Theoretically, the parameters of the columns are fitted independently, which is equivalent to decomposing the structured light surface into multiple single-point line lasers (structured light emitted by the single-point line laser emitter). For example, the structured light (single-line structured light) is divided into single-point line lasers in multiple directions according to a preset angle (just like a straight line laser is divided into multiple single-point line lasers), and then the triangulation method (pinhole imaging principle) is used for each single-point line laser to speed up the calibration, so that the triangulation method in different directions or different test angles or different calibration columns can calibrate the corresponding modeling parameters.

[0084] Preferably, the calibration device further comprises a stepper motor, the calibration plate is mounted at one end of the slide rail, and the structured light module is mounted at the other end of the slide rail, then the stepper motor can drive the calibration plate to move away from the baseline of the structured light module (the line between the center of the structured light emitter and the center of the camera) at a certain step, or to move closer to the baseline of the structured light module (the line between the center of the structured light emitter and the center of the camera) at a certain step. Specifically, the chip controls the stepper motor to drive the calibration plate to slide on the slide rail; each time the calibration plate slides a fixed distance along the slide rail, the camera captures the calibration plate image once, the imaging position of the light bar in the imaging plane is offset by a preset height along the height direction, and the corresponding projected light bar center coordinate is offset by a longitudinal offset along the height direction in the pixel plane. Therefore, the calibration model with the structured light emitter as the main perspective obtains the light bar center coordinate and its offset required to construct the structured light ranging relationship on the same light bar, the offset of the imaging position of the light bar in the imaging plane, and the distance the calibration plate slides along the slide rail.

[0085] Therefore, in this embodiment, a motor, a calibration plate, a slide rail, and a clamp for fixing the sensor in the structured light module constitute a calibration device for calibrating the structured light module under the drive of the motor, wherein the structured light module can collect two-dimensional images of the calibration plate at different distances, and the light strips in a certain frame of the two-dimensional image will be clearly imaged in the imaging plane of the camera and will not deviate from the imaging plane of the camera (the surface of the photosensitive element). When the structured light module changes, including at least one of the focal length of the camera, the relative position between the structured light emitter and the camera, the emission direction of the structured light emitter (its change will affect whether a light strip can be projected on the calibration plate at a certain distance), and the optical axis direction of the camera, the calibration plate needs to be driven by a motor to slide to the corresponding position before the structured light emitted by the structured light emitter can be incident on the calibration plate and the reflected light can be imaged through the lens of the camera to the surface of the camera's photosensitive element. Only then can the camera capture the image containing the light strip, and then the camera can be calibrated. Therefore, it is necessary to constrain the sliding distance of the calibration plate on the slide rail, such as constraining the calibration plate to slide only one step distance each time under the drive of the motor to meet the camera imaging requirements. Preferably, in the case where the camera is a distortion-free camera, when the surface of the camera's photosensitive element is at a certain oblique angle to the measured surface of the calibration plate, the distance between the baseline and the calibration plate can be directly calculated according to the pinhole imaging principle, which will not affect the calibration accuracy of the camera.

[0086] In summary, the calibration plate calibrates the structured light module under the drive of the motor. There is no need to specially set up targets such as a checkerboard. The calibration can be performed by simply controlling the calibration plate to slide along the slide rail under the control of the motor, which is easy to automate. This improves the efficiency of the structured light module calibration while ensuring the reliability of the calibration results of the structured light module.

[0087] Preferably, the calibration plate is arranged on the normal plane of the slide rail; the row pixel position in the pixel plane is parallel to the placement plane of the slide rail so that the line connecting a row of pixel points corresponding to the light strip is parallel to the placement plane of the slide rail, and the column pixel position in the pixel plane is regarded as perpendicular to the slide rail, wherein the slide rail can be placed on a horizontal plane; the structured light emitted by the structured light emitter is a straight-line structured light. Therefore, in the process of the calibration plate sliding along the slide rail, the intersection line between the structured light emitted by the structured light emitter and the calibration plate (the light strip) can be parallel to the horizontal plane, and the optical axis direction of the camera can be parallel to the slide rail. This realizes calibration for the height direction disclosed in the aforementioned embodiment while reducing coordinate system conversion or angle conversion, thereby improving the convenience of calibration.

[0088] Preferably, the structured light emitted by the structured light emitter is vertically incident on the measured surface of the calibration plate, and the intersection line between the structured light and the calibration plate is parallel to the horizontal line; specifically, when the slide rail is placed on a horizontal plane and the calibration plate is located on the normal plane of the slide rail (the normal plane in the direction parallel to the slide rail), under the fixed assembly of the fixture, after the structured light surface emitted by the structured light emitter is vertically incident on the measured surface of the calibration plate, the intersection line (light strip) between the structured light surface and the measured surface of the calibration plate is parallel to the horizontal plane, and the row pixel position in the pixel plane is parallel to the placement plane of the slide rail. Reduce the conversion of coordinate system angles and speed up the construction of structured light ranging relationship using the pinhole imaging principle or use the least squares method for fitting processing.

[0089] Preferably, the Z axis of the camera coordinate system is perpendicular to the measured surface of the calibration plate, and the intersection line of the structured light and the calibration plate is parallel to the horizontal line. Specifically, when the slide rail is placed on a horizontal plane and the calibration plate is located on the normal plane of the slide rail (the normal plane in the direction parallel to the slide rail), under the fixed assembly of the fixture, the Z axis of the camera coordinate system is perpendicular to the measured surface of the calibration plate (if the imaging plane of the camera is set on the normal plane of the slide rail (the normal plane in the direction parallel to the slide rail), the Z axis of the camera coordinate system is regarded as the optical axis), the intersection line (light bar) of the structured light plane and the measured surface of the calibration plate is parallel to the horizontal plane, and the row pixel position in the pixel plane is parallel to the placement plane of the slide rail. Reduce the conversion of coordinate system angles and speed up the construction of structured light ranging relationship using the pinhole imaging principle or use the least squares method for fitting.

[0090] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable technicians familiar with the technical field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for calibrating modeling parameters of a structured light module. It is characterized in that The structured light module includes a structured light emitter and a camera; the modeling parameter calibration method includes the following steps: Step 1, control the structured light emitter to emit structured light, project it onto the calibration plate to form light strips; at the same time, control the calibration plate to move relative to the structured light module, and the camera collects the calibration plate images at different test distances to obtain multiple frames of calibration plate images, and extracts the center coordinates of the light strips of each calibration column from the same frame of the calibration plate image, and determines the test distance corresponding to the same calibration column of the same frame of the calibration plate image; each frame of the calibration plate image includes the light strips; Step 2: In the multiple frames of calibration plate images obtained in step 1, nonlinear fitting is performed on the center coordinates of the light strips in the same calibration column and the test distance corresponding to the same calibration column to obtain the modeling parameters corresponding to the same calibration column; Step 3, repeatedly executing step 2 to traverse the light bar center coordinates of each calibration column in each frame of the calibration plate image obtained in step 1 and the corresponding test distance, obtain the modeling parameters corresponding to all calibration columns in turn, and determine that the calibration of the modeling parameters of the structured light module is completed; Among them, the modeling parameters corresponding to the same calibration column are the coefficients required to construct the structured light ranging relationship under the same calibration column. The structured light ranging relationship is the relationship between the center coordinates of the light strips in the same calibration column and their corresponding test distances constructed based on the pinhole imaging principle.

2. According to the modeling parameter calibration method of claim 1, It is characterized in that The pixel positions of the light spots of the light strip at each test angle projected in the photosensitive element of the camera are marked as pixel positions in the corresponding calibration columns of the pixel plane, so that the light strip is represented by pixel points of multiple calibration columns in the pixel plane; each calibration column corresponds to a test angle.

3. According to the modeling parameter calibration method of claim 2, It is characterized in that Before executing step 1, the emission direction of the structured light emitter is configured to be set at an angle with the optical axis direction of the camera, and the line between the center of the structured light emitter and the optical center of the camera is set to be parallel to the measured surface of the calibration plate; the structured light emitted by the structured light emitter is set to intersect with the measured surface of the calibration plate, so that the structured light emitted by the structured light emitter is incident on the calibration plate at a certain angle to the normal line of the measured surface of the calibration plate, and the reflected light is imaged into the photosensitive element of the camera through the lens of the camera; The camera collects the imaging of the reflected light of the structured light in the photosensitive element of the camera, which is represented by the imaging of the light strip in the imaging plane of the camera; the imaging of the light strip in the imaging plane of the camera is a part of the calibration plate image, so that the calibration plate image forms a two-dimensional structured light stripe image.

4. According to the modeling parameter calibration method of claim 3, It is characterized in that In step 1, the method for controlling the displacement of the calibration plate relative to the structured light module includes driving the calibration plate to slide on a slide rail, wherein the structured light module is fixedly mounted at one end of the slide rail, and the initial position of the calibration plate is set at the other end of the slide rail; Each time the calibration plate slides a certain distance along the slide rail, the camera captures an image of the calibration plate, extracts the center coordinates of the light bar formed by the light bar in the pixel plane from the calibration plate image, and records the test distance at the corresponding test angle.

5. According to the modeling parameter calibration method of claim 4, It is characterized in that In step 1, when the calibration plate slides along the slide rail from the upper limit of the range to the lower limit of the range, or when the calibration plate slides along the slide rail from the lower limit of the range to the upper limit of the range, each time the calibration plate slides a preset fixed distance, the camera captures the image of the calibration plate, the imaging position of the light bar in the imaging plane is offset by a preset height in the height direction, and the corresponding central pixel coordinate of the projected light bar is offset once in the opposite direction of the height direction in the pixel plane; The offset formed by the imaging position of the light strip in the imaging plane relative to the reference position is marked as the light spot offset.

6. The modeling parameter calibration method according to claim 5, It is characterized in that The reference position is the imaging position of the reflected light of the structured light emitted by the structured light emitter on the measured surface of the measured object farthest from the structured light module in the detectable area of ​​the structured light module in the camera; Wherein, the object to be measured includes the calibration plate.

7. The modeling parameter calibration method according to claim 6, It is characterized in that The upper limit of the range of the slide rail is the position farthest from the structured light module that the calibration plate is allowed to slide on the slide rail, and is within the detectable area of ​​the structured light module; The lower limit of the range of the slide rail is the position of the calibration plate that is allowed to slide on the slide rail and is closest to the structured light module, and is within the detectable area of ​​the structured light module.

8. The modeling parameter calibration method according to claim 7, It is characterized in that Under the same calibration column, the methods for constructing the structured light ranging relationship include: The line between the center of the structured light emitter and the optical center of the camera is set as the baseline, and the length of the baseline is set as the baseline distance; Set the distance between the baseline and the calibration plate as the test distance; The constructed structured light ranging relationship is: Px=k0 / d+k1, where d is the test distance and Px is the center coordinate of the light bar; The coefficients required to construct the structured light ranging relationship under the same calibration column are k0 and k1 respectively; Under the same calibration column or the same test angle, the ratio of the baseline distance to the test distance is equal to the ratio of the light spot offset to the focal length of the camera; the light spot offset is equal to the sum of the product of the center coordinates of the light bar and the size of a single pixel and the projection error; the imaging of the light bar in the imaging plane of the camera is set in the imaging plane coordinate system, and the reference position and the light spot offset are both set in the imaging plane coordinate system; the center coordinates of the light bar are set in the pixel coordinate system, and the pixel coordinate system is the coordinate system of the pixel plane of the camera.

9. The modeling parameter calibration method according to claim 8, It is characterized in that In step 2, the method of performing nonlinear fitting on the center coordinates of the light bars of the same calibration column and the test distance corresponding to the same calibration column includes: Step 21, sequentially traverse the i-th calibration column of the M frames of calibration plate images acquired in step 1; Step 22, extract the row pixel position coordinate Pxj of the center of the light strip of the i-th calibration column from the currently traversed frame of the calibration plate image, and record it as the center coordinate of the light strip of the i-th column of the currently traversed frame of the calibration plate image, and at the same time determine the test distance dj corresponding to the i-th calibration column of the same frame of the calibration plate image; j represents the sorting number of the currently traversed frame of the calibration plate image, so that the currently traversed frame of the calibration plate image is marked as the j-th frame of the calibration plate image; j is a positive integer, j is greater than 0, and j is less than or equal to M; Step 23, configure j=j+1, so that the data of the j+1th calibration plate image is updated to the data of the jth calibration plate image, and then repeat step 22 until the i-th calibration column of the M-frame calibration plate image is traversed, and the i-th column light bar center coordinates in each frame calibration plate image are combined into a coordinate set {Px1, Px2, ..., PxM}, and at the same time, the test distance corresponding to the i-th calibration column in each frame calibration plate image is combined into a distance set {d1, d2, ..., dM} or a distance set {1 / d1, 1 / d2, ..., 1 / dM}; Step 24, configure the coordinate set of step 23 and the distance set of step 23 as data sets on two coordinate axes, perform nonlinear fitting, obtain the corresponding modeling parameters k0 and modeling parameters k1 of the i-th calibration column, and determine the structured light ranging relationship corresponding to the i-th calibration column; and record the modeling parameters k0 and modeling parameters k1 corresponding to the i-th calibration column as a modeling parameter pair of the i-th calibration column; then execute step 3.

10. The modeling parameter calibration method according to claim 9, It is characterized in that In step 3, i=i+1 is configured so that the pixels of the i+1th calibration column in the jth calibration plate image are updated to the pixels of the ith calibration column in the jth calibration plate image, and steps 21 to 24 are repeated until the N calibration columns in each calibration plate image acquired in step 1 are traversed, and after all calibration columns are determined to pass nonlinear fitting, N modeling parameter pairs are accumulated, and it is determined that the calibration of the modeling parameters of the structured light module is completed; In the M frames of calibration plate images, by repeatedly executing steps 21 to 24, N structured light ranging relationship equations are cumulatively fitted; The N modeling parameter pairs correspond to N calibration columns in each frame of the calibration plate image, respectively. The N calibration columns are N columns of the same frame of the calibration plate image that are calibrated in the pixel plane of the camera, and each column corresponds to one of the test angles and one of the test distances. i represents the column number of a calibration column currently traversed in a frame of calibration plate image, i is a positive integer, i is greater than 0, i is less than or equal to N.

11. The modeling parameter calibration method according to claim 10, It is characterized in that The least square method is used for nonlinear fitting to minimize the fitting error.

12. A chip, It is characterized in that The chip is used to execute the modeling parameter calibration method described in any one of claims 1 to 11.

13. A calibration device, It is characterized in that The calibration device includes a calibration plate, a slide rail and the chip described in claim 12, and the calibration device is used to calibrate the structured light module described in any one of claims 1 to 11, wherein the chip is used to control the calibration plate to slide along the slide rail, and control the structured light emitter to emit structured light to the measured surface of the calibration plate, and at the same time control the camera to collect the image of the structured light projected onto the measured surface of the calibration plate, so as to obtain the modeling parameters corresponding to each calibration column in each frame of the calibration plate image by executing steps 1 to 3, and determine that the calibration of the modeling parameters of the structured light module is completed; wherein the structured light module includes a structured light emitter and a camera.

14. The calibration device according to claim 13, It is characterized in that The calibration device also includes a fixture, which is used to fix the line between the center of the structured light emitter and the optical center of the camera to be parallel to the measured surface of the calibration plate, and is also used to assemble the emission direction of the structured light emitter and the optical axis direction of the camera to form an angle; the distance between the center of the structured light emitter and the optical center of the camera is fixed; The emission direction of the structured light emitter intersects with the calibration plate, so that the structured light emitted by the structured light emitter is incident on the calibration plate at a certain angle to the normal line of the measured surface of the calibration plate, and the reflected light is imaged into the photosensitive element of the camera through the lens of the camera; The camera is configured to avoid distortion.

15. The calibration device according to claim 13, It is characterized in that The calibration device also includes a stepping motor, the calibration plate is slidably mounted on one end of the slide rail, and the structured light module is fixedly mounted on the other end of the slide rail; The chip controls the stepper motor to drive the calibration plate to slide on the slide rail; each time the calibration plate slides a fixed distance along the slide rail, the camera captures an image of the calibration plate, and the corresponding center coordinate of the projected light bar is offset by a preset longitudinal offset in the height direction in the pixel plane.

16. The calibration device according to claim 13, It is characterized in that The measured surface of the calibration plate is arranged on the normal plane of the slide rail, and the row pixel positions in the pixel plane are parallel to the placement plane of the slide rail; The structured light emitted by the structured light emitter is a straight-line structured light.

17. The calibration device according to claim 13, It is characterized in that The structured light emitted by the structured light emitter is vertically incident on the measured surface of the calibration plate, and the intersection line of the structured light and the calibration plate is parallel to the horizontal line; or, the Z axis of the camera coordinate system is perpendicular to the measured surface of the calibration plate, and the intersection line of the structured light and the calibration plate is parallel to the horizontal line.

Citation Information

Patent Citations

  • Multi-line structure light sensor calibration method

    CN106441099A

  • Light-sectioning data modeling and reconstruction method fusing planar mesh depth calculation

    CN108596929A

  • Design method of pavement flatness detection equipment

    CN114232434A

  • Structural Light Parameter Calibration Device and Method Based on Front-Coating Plane Mirror

    US20200132451A1

  • Calibration method and calibration device for line structured light measurement system, and system

    WO2022143796A1