Image reconstruction method, device and equipment

Abstract

An image reconstruction method, device, and instrument are provided. The image reconstruction method includes the steps of: acquiring a first original image collected by a first camera and a second original image collected by a second camera when a multi-line laser projects N lines of structured light onto an object to be measured (201); determining a first target image corresponding to the first original image and a second target image corresponding to the second original image (202); and determining a first light strip centerline corresponding to each first light-emitting area in the first target image and a second light strip centerline corresponding to each second light-emitting area in the second target image. The method includes: determining (203) a plurality of keypoint pairs based on a first light strip centerline and a second light strip centerline, the keypoint pair including a first pixel point at the first light strip centerline and a second pixel point at the second light strip centerline; determining (205) three-dimensional points based on the keypoint pairs and camera calibration parameters; and generating (206) a three-dimensional reconstructed image based on the three-dimensional points corresponding to the plurality of keypoint pairs.

Description

[Technical field]

[0001] The present invention relates to the technical field of image processing, and in particular to an image reconstruction method, device and apparatus. [Background technology]

[0002] The three-dimensional imaging device includes a laser and a camera. The laser is used to project a line structured light onto the surface of the object to be measured (i.e., the object to be measured), and the camera is used to photograph the object to be measured and obtain an image having the line structured light, i.e., a line structured light image. After obtaining the line structured light image, the light strip center line of the line structured light image can be obtained, and the light strip center line can be transformed based on the predetermined sensor parameters to obtain the spatial coordinates (i.e., three-dimensional coordinates) of the object to be measured at the current position. Based on the spatial coordinates of the object to be measured at the current position, a three-dimensional reconstruction of the object to be measured can be realized. Summary of the Invention [Means for solving the problem]

[0003] An embodiment of the present invention provides an image reconstruction method applied to a three-dimensional imaging device, the three-dimensional imaging device including a first camera, a second camera, and a multi-line laser, the method comprising: When the multi-line laser projects N lines of structured light onto the object, acquiring a first original image of the object collected by the first camera and acquiring a second original image of the object collected by the second camera, where N is a positive integer greater than 1; determining a first target image corresponding to the first original image and a second target image corresponding to the second original image, the first target image including N first light-emitting regions corresponding to the N line structured lights, and the second target image including N second light-emitting regions corresponding to the N line structured lights; determining a first light strip centerline corresponding to each first light emitting area in the first target image and determining a second light strip centerline corresponding to each second light emitting area in the second target image; determining a plurality of keypoint pairs based on all of the first light strip centerlines and all of the second light strip centerlines, where for each keypoint pair, the keypoint pair includes a first pixel point on the first light strip centerline and a second pixel point on the second light strip centerline, the first pixel point and the second pixel point being pixel points corresponding to a same location point on the object; determining 3D points corresponding to the keypoint pairs based on the keypoint pairs and camera calibration parameters; and generating a 3D reconstructed image of the object based on 3D points corresponding to the plurality of key point pairs.

[0004] An embodiment of the present invention provides an image reconstruction apparatus for application to a three-dimensional imaging apparatus, the three-dimensional imaging apparatus including a first camera, a second camera, and a multi-line laser, the apparatus comprising: an acquisition module configured to acquire a first original image of the object collected by the first camera and acquire a second original image of the object collected by the second camera when the multi-line laser projects N lines of structured light onto the object, where N is a positive integer greater than 1; a determination module configured to: determine a first target image corresponding to the first original image and a second target image corresponding to the second original image; determine first light strip centerlines corresponding to each first light emitting area in the first target image; determine second light strip centerlines corresponding to each second light emitting area in the second target image; determine a plurality of keypoint pairs based on all the first light strip centerlines and all the second light strip centerlines; and determine three-dimensional points corresponding to the keypoint pairs based on the keypoint pairs and camera calibration parameters; a generation module configured to generate a 3D reconstruction of the object based on 3D points corresponding to the plurality of key point pairs; The first target image includes N first light-emitting regions corresponding to the N line structured lights, and the second target image includes N second light-emitting regions corresponding to the N line structured lights; For each key point pair, the key point pair includes a first pixel point at the first light strip centerline and a second pixel point at the second light strip centerline, where the first pixel point and the second pixel point correspond to the same location point on the object to be measured.

[0005] An embodiment of the present invention provides a three-dimensional imaging device including a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor, the processor being configured to execute the image reconstruction method disclosed in the above embodiment of the present invention by executing the machine-executable instructions. Effect of the Invention

[0006] In the embodiment of the present invention, the multi-line laser projects N line structured lights onto the object each time, where N is a positive integer greater than 1, such as 7, 11, 15, etc., so that the line structured light image collected by the camera each time includes N light strip center lines. The line structured light image is equivalent to the line structured light images of N positions on the object to be measured. This can reduce the number of collections of the line structured light images and reduce the time for three-dimensional reconstruction. When scanning the surface of the object to be measured with the multi-line laser, the entire contour data of the object to be measured can be quickly obtained, and the three-dimensional image information of the object to be measured can be output, improving the detection accuracy and detection speed. In addition, the first camera and the second camera are used to simultaneously collect line structured light images, and based on the line structured light images collected by the two cameras, three-dimensional information of the object to be measured can be obtained by triangulation, that is, depth information of the object to be measured can be obtained, so that the depth information of the multi-line laser light can be obtained by a single collected image, the efficiency of a single scan can be improved by N times, and a full-size scan of the entire contour of the object to be measured can be quickly realized. [Brief description of the drawings]

[0007] [Figure 1A] 1 is a schematic configuration diagram of a three-dimensional imaging device according to an embodiment of the present invention. [Figure 1B] 1 is a schematic configuration diagram of a three-dimensional imaging device according to an embodiment of the present invention. [Figure 1C] FIG. 2 is a schematic diagram of a multiple line laser according to an embodiment of the present invention. [Figure 1D] FIG. 2 is a schematic diagram of a multiple line laser according to an embodiment of the present invention. [Diagram 2] 1 is a flowchart of an image reconstruction method according to an embodiment of the present invention. [Diagram 3] 1 is a flowchart of an image reconstruction method according to an embodiment of the present invention. [Figure 4] 1 is a schematic diagram illustrating the principle of a triangulation method according to an embodiment of the present invention; [Diagram 5]1 is a flowchart of an image reconstruction method according to an embodiment of the present invention. [Figure 6] 1 is a schematic configuration diagram of an image reconstruction device according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The terms used in the embodiments of the present invention are not intended to limit the present invention, but merely to describe specific embodiments. The singular forms "a," "the," and "the" used in the embodiments and claims of the present invention are also intended to include the plural, unless the context clearly indicates otherwise. The term "and / or" as used herein should be understood to mean any or all possible combinations including one or more of the associated listed items.

[0009] In the examples herein, terms such as first, second, third, etc. may be used to describe various pieces of information, but it should be understood that these pieces of information should not be limited to these terms. These terms are used only to distinguish between the same types of information. For example, a first piece of information may be referred to as a second piece of information, and similarly, a second piece of information may be referred to as a first piece of information, without departing from the scope of the present invention. Depending on the context, furthermore, the word "if" used may be interpreted as "when...", or "when...", or "in response to determining...".

[0010] In order to obtain a three-dimensional reconstruction image, the related art can adopt surface structured light projection, binocular speckle method, TOF (Time of flight) method and single-line laser contour scanning method, etc. When adopting surface structured light projection, DLP (Digital Light Processing) or LCD (Liquid Crystal Display) projection technology can be adopted, and LED (Light-Emitting Diode) light source can be used as the projection light source, which has a large projection volume, energy dispersion, large volume with a large field of view at a long distance, and high power consumption, which is disadvantageous for the application of three-dimensional positioning. When adopting binocular speckle method, a method combining binocular parallax and laser speckle binocular matching method is adopted, which has low detection accuracy and poor edge contour, which is disadvantageous for the application of contour scanning and three-dimensional positioning. When adopting TOF method, it is limited by the resolution of the camera, and the detection accuracy is at the centimeter level, which is not suitable for the application of automated high-precision positioning. When adopting the single-line laser contour scanning method, the depth information of the object is scanned by a single-line laser, so the scanning speed is slow and the stability is poor, and the positioning needs of 3D reconstruction cannot be met.

[0011] Taking the single-line laser contour scanning method as an example, a laser is used to project a line structured light onto the surface of the object to be measured, and a camera is used to photograph the object to be measured, so that an image having line structured light, that is, a line structured light image, can be obtained. After the line structured light image is obtained, the spatial coordinates (i.e., three-dimensional coordinates) of the object to be measured at the current position can be obtained based on the line structured light image, and three-dimensional reconstruction of the object to be measured can be realized. However, in order to realize the three-dimensional reconstruction of the object to be measured, it is necessary to collect line structured light images of different positions of the object to be measured. That is, the laser projects the line structured light onto different positions of the object to be measured, and each position corresponds to one line structured light image. Since the camera only collects a line structured light image corresponding to one position each time, it is necessary to collect the line structured light image multiple times to complete the three-dimensional reconstruction. That is, the time of the three-dimensional reconstruction is relatively long, the scanning speed is slow, and the stability is poor, so that the positioning needs of the three-dimensional reconstruction are not met.

[0012] In view of this, an embodiment of the present invention provides a 3D imaging method using multi-line laser scanning, which uses triangulation to obtain depth information of an object to be measured, and uses a multi-line laser to form an optical scan on the surface of the object to quickly obtain the entire contour data of the object to be measured, and outputs 3D image information of the object to be measured. Here, the 3D imaging method using multi-line laser scanning can be applied to the machine vision field and the industrial automation field, and can be used to realize 3D measurement and robot positioning, and its application scenario is not limited.

[0013] In this embodiment, the depth information of the object to be measured may be obtained by using triangulation. By obtaining the depth information of the multi-line laser light of 10 lines, 20 lines, etc. once, the efficiency of one scan can be improved by 10 to 20 times, and then, by performing the multi-line laser light scanning, the full-size scanning of the entire contour of the object to be measured can be realized.

[0014] In this embodiment, the problem of large volume and large power consumption of the surface structured light projection method is solved, the problem of low detection accuracy and poor edge contour of the binocular speckle method is solved, the problem of low detection accuracy of the TOF method is solved, and the problem of slow scanning speed and poor stability of the single-line laser contour scanning method is solved. From the above, the three-dimensional imaging method using multi-line laser scanning in this embodiment is a three-dimensional scanning imaging method with faster detection speed and higher detection accuracy, high precision, low cost, small volume and low power consumption.

[0015] An embodiment of the present invention provides a 3D imaging method by multi-line laser scanning, which may be applied to a 3D imaging device, which may be any device having 3D imaging function, such as any device in the machine vision field or industrial automation field, and the type of the 3D imaging device is not particularly limited.

[0016] As shown in FIG. 1A, a schematic diagram of a practical configuration of a three-dimensional imaging device, which may include, but is not limited to, a left camera, a right camera, an auxiliary camera, a processor, a laser, a galvanometer motor, and a galvanometer drive. In another configuration of the three-dimensional imaging device, the three-dimensional imaging device may include, but is not limited to, a left camera, a right camera, a processor, a laser, a galvanometer motor, and a galvanometer drive, i.e., does not have an auxiliary camera.

[0017] As shown in FIG. 1B, a block diagram of a configuration of a three-dimensional imaging apparatus is another representation of the three-dimensional imaging apparatus, which may include, but is not limited to, an image acquisition device 100, an optical-mechanical scanning device 200, a multi-line laser emission device 300, an image processing device 400, and a fixed bracket 500.

[0018] Exemplarily, the image acquisition device 100 may include a left camera 101 and a right camera 102, and the image acquisition device 100 may include a left camera 101, a right camera 102, and an auxiliary camera 103. The left camera 101 and the right camera 102 acquire depth information based on a triangulation method by a binocular matching multi-line laser. The auxiliary camera 103 may or may not be involved in the reconstruction. Here, the left camera 101 and the right camera 102 may use a black and white camera, and a filter with the same bandwidth as the laser wavelength is added to the tip of the camera to pass only light within the laser wavelength range. That is, by receiving only the light of the laser wavelength reflected on the surface of the measured object and acquiring a reflected image of the laser beam, it is possible to improve the contrast and reduce the interference of environmental light. The left camera 101 and the right camera 102 may be disposed on both sides of the mechanical galvanometer, respectively, and installed symmetrically. The auxiliary camera 103 may use a black and white camera or an RGB camera. The auxiliary camera 103 is mounted as close as possible to the output optical axis of the mechanical galvanometer, thereby ensuring a short baseline and making the field of view and the laser scanning field of view nearly coincident, thus the auxiliary camera 103 can completely capture all the laser beams every time. If the auxiliary camera 103 adopts an RGB camera, it can turn off the laser and capture a color image of the surface of one piece of the measured object, thereby realizing the output function of the RGBD image.

[0019] Exemplarily, the opto-mechanical scanning device 200 may include a mechanical galvanometer. That is, the scanning function is realized using a mechanical galvanometer. The mechanical galvanometer may include three parts: a galvanometer motor, a galvanometer drive, and a mirror. In FIG. 1A, only the galvanometer motor and the galvanometer drive are shown. The mechanical galvanometer may be a mechanical galvanometer with high repeatability. The reflecting mirror has a visible light reflecting film, which can reflect the laser beam and change the emission angle of the laser beam.

[0020] Exemplarily, the multi-line laser emission device 300 may include a laser. The laser may be a multi-line laser, i.e., a laser that simultaneously emits multiple laser beams. For example, the laser may be a multi-line laser module mainly composed of a laser diode, a collimating lens, and a multi-line DOE (Diffractive Optical Element). The laser diode adopts a high-power red laser diode, and the wavelength may be 635 nm, 660 nm, or other wavelengths. The multi-line DOE may simultaneously emit 10 laser beams, 11 laser beams, or 25 laser beams, etc., but is not limited thereto.

[0021] Exemplarily, the image processing device 400 may include a processor such as a CPU or a GPU. The image processing device 400 is connected to the image acquisition device 100, the optical mechanical scanning device 200, and the multi-line laser emitting device 300, respectively. The image processing device 400 activates the multi-line laser emitting device 300, and causes the multi-line laser emitting device 300 to emit a plurality of line lasers (i.e., line structured light), and the plurality of line lasers are irradiated on the surface of the object to be measured after being reflected by the mechanical galvanometer. The image processing device 400 controls the mechanical galvanometer to start scanning, and each time one angle is scanned, the image processing device 400 may receive angle feedback information of the mechanical galvanometer, and trigger the image acquisition device 100 based on the angle feedback information to collect a multi-line laser image of the surface of the object to be measured.

[0022] For ease of explanation, in the following embodiment, the image processing device 400 is a processor, the image acquisition device 100 is a left camera, a right camera and an auxiliary camera, the optical mechanical scanning device 200 is a mechanical galvanometer, and the multi-line laser emitting device 300 is a laser. As shown in FIG. 1C, the laser can emit multiple line lasers (FIG. 1C shows seven line lasers as an example), and these line lasers are irradiated on the surface of the object to be measured after being reflected by the mechanical galvanometer. The initial angle of the mechanical galvanometer is angle A, and at angle A, the left camera collects a line structured light image A1 of the object to be measured, the right camera collects a line structured light image A2 of the object to be measured, and the auxiliary camera collects a line structured light image A3 of the object to be measured. Then, as shown in FIG. 1D, the angle of the mechanical galvanometer is angle B, the left camera collects a line structured light image B1 of the object to be measured, the right camera collects a line structured light image B2 of the object to be measured, and the auxiliary camera collects a line structured light image B3 of the object to be measured, and so on until the final angle of the mechanical galvanometer is reached, indicating that a complete scan of the surface of the object to be measured is completed.

[0023] Based on the line structured light image A1, the line structured light image A2, and the line structured light image A3, the processor can determine a three-dimensional point (i.e., a three-dimensional point cloud) at an angle A. Based on the line structured light image B1, the line structured light image B2, and the line structured light image B3, the processor can determine a three-dimensional point at an angle B. By analogy in this way, three-dimensional points at all angles can be obtained. Based on this, the three-dimensional points at all angles can be stitched together to obtain a complete three-dimensional point of the surface of the object to be measured, that is, a complete three-dimensional reconstruction image of the surface of the object to be measured can be obtained.

[0024] If the auxiliary camera adopts an RGB camera, after completing the scan, the processor may turn off the laser, control the auxiliary camera to collect an RGB image, and then fit and align the three-dimensional reconstruction image and the RGB image to output an RGBD image. The processing process of the RGBD image is not limited in this embodiment.

[0025] For example, the fixing bracket 500 plays the role of fixing and heat dissipation, and adopts metal aluminum or other materials. The camera and the mechanical galvanometer adopt an integrated fixing design, which ensures that the relative position between the camera and the mechanical galvanometer does not change.

[0026] Hereinafter, the image reconstruction method according to the embodiment of the present invention will be described with reference to a specific embodiment. FIG 2 is a flowchart of the image reconstruction method according to the embodiment of the present invention, and the method may include steps 201 to 206.

[0027] In step 201, when a multi-line laser projects N lines of structured light (i.e., laser beams) onto a measurement object (i.e., an object to be measured), a first original image of the measurement object is acquired by a first camera, and a second original image of the measurement object is acquired by a second camera, where N may be a positive integer greater than 1.

[0028] Exemplarily, the image reconstruction method may be applied to a three-dimensional imaging device. The three-dimensional imaging device may include a first camera, a second camera, and a multi-line laser. The three-dimensional imaging device may further include a processor and a mechanical galvanometer. Here, the first camera may be a left camera, and the second camera may be a right camera. Or, the first camera may be a right camera, and the second camera may be a left camera. The multi-line laser is the laser of the above embodiment.

[0029] When the angle of the mechanical galvanometer is angle A, the multi-line laser projects N lines of line structured light onto the object to be measured, the left camera collects a line structured light image A1 of the object to be measured, and the right camera collects a line structured light image A2 of the object to be measured. Based on this, assuming that the first camera is the left camera and the second camera is the right camera, the processor may obtain a first original image (e.g., the line structured light image A1) and a second original image (e.g., the line structured light image A2), and perform subsequent processing based on the first original image and the second original image.

[0030] When the angle of the mechanical galvanometer is angle B, the multi-line laser projects N lines of structured light onto the object to be measured, the left camera collects a line structured light image B1 of the object to be measured, and the right camera collects a line structured light image B2 of the object to be measured. Based on this, the processor acquires a first original image (line structured light image B1) and a second original image (line structured light image B2), and performs subsequent processing based on the first original image and the second original image. In this way, at each angle of the mechanical galvanometer, the processor may acquire a first original image of the object to be measured collected by the first camera, and a second original image of the object to be measured collected by the second camera.

[0031] For example, to project N line structured lights onto the object by a multi-line laser, the first original image may include N first light-emitting areas corresponding to the N line structured lights. The N first light-emitting areas have a one-to-one correspondence with the N line structured lights. The second original image may include N second light-emitting areas corresponding to the N line structured lights. The N second light-emitting areas have a one-to-one correspondence with the N line structured lights.

[0032] In step 202, a first target image corresponding to the first original image and a second target image corresponding to the second original image are determined. The first target image includes N first light-emitting regions corresponding to the N line structured lights, and the second target image includes N second light-emitting regions corresponding to the N line structured lights.

[0033] In one possible embodiment, the first original image may be determined as the first target image, and the second original image may be determined as the second target image. Or, the first original image and the second original image are subjected to binocular correction, and the first target image corresponding to the first original image and the second target image corresponding to the second original image are obtained. Here, the binocular correction is used to make the same position point on the object to be measured have the same pixel height in the first target image and the second target image, and this binocular correction process is not particularly limited.

[0034] In step 203, a first light strip centerline corresponding to each first light-emitting area in the first target image is determined, and a second light strip centerline corresponding to each second light-emitting area in the second target image is determined.

[0035] In step 204, a plurality of keypoint pairs are determined based on all the first light strip centerlines and all the second light strip centerlines, where each keypoint pair includes a first pixel point on the first light strip centerline and a second pixel point on the second light strip centerline, the first pixel point and the second pixel point corresponding to the same location point on the object.

[0036] In one possible embodiment, for each line structured light, a target first light strip centerline and a target second light strip centerline corresponding to the line structured light may be determined from all the first light strip centerlines and all the second light strip centerlines, and for each first pixel point on the target first light strip centerline, project the first pixel point onto a second target image to obtain a projected pixel point corresponding to the first pixel point, select a second pixel point from the target second light strip centerline corresponding to the projected pixel point, and generate a key point pair based on the first pixel point and the second pixel point, i.e., the key point pair includes the first pixel point and the second pixel point.

[0037] Exemplarily, selecting a second pixel point corresponding to the projected pixel point from the target second light strip centerline includes, but is not limited to, determining a pixel point from the target second light strip centerline having the same pixel height as the projected pixel point, and if the determined pixel point is one, selecting the pixel point as the second pixel point; if the determined pixel points are at least two, determining reprojection errors between the at least two pixel points and the projected pixel point, and selecting the pixel point corresponding to the smallest reprojection error as the second pixel point.

[0038] Exemplarily, projecting a first pixel point onto a second target image and obtaining a projected pixel point corresponding to the first pixel point includes, but is not limited to, obtaining a first calibration equation and a second calibration equation corresponding to the line structured light, converting the first pixel point into a target three-dimensional reconstruction point based on the first calibration equation, and converting the target three-dimensional reconstruction point into the projected pixel point based on the second calibration equation, where the first calibration equation represents a functional relationship between the pixel point in the first target image and the three-dimensional reconstruction point, and the second calibration equation represents a functional relationship between the pixel point in the second target image and the three-dimensional reconstruction point.

[0039] In step 205, a 3D point corresponding to the keypoint pair is determined based on the keypoint pair and the camera calibration parameters.

[0040] In one possible embodiment, the camera calibration parameters may include the internal camera parameters of the first camera, the internal camera parameters of the second camera, and the external camera parameters between the first camera and the second camera. The first pixel point may be corrected for distortion by the internal camera parameters of the first camera, and the pixel point after distortion correction may be transformed into a first homogeneous coordinate, and the second pixel point may be corrected for distortion by the internal camera parameters of the second camera, and the pixel point after distortion correction may be transformed into a second homogeneous coordinate. Then, a three-dimensional point corresponding to the key point pair is determined by a triangulation method based on the first homogeneous coordinate, the second homogeneous coordinate, the internal camera parameters of the first camera, the internal camera parameters of the second camera, and the external camera parameters. The triangulation method is not limited.

[0041] In step 206, a 3D reconstruction of the object is generated based on the 3D points corresponding to the plurality of keypoint pairs.

[0042] For example, when the angle of the mechanical galvanometer is angle A, three-dimensional points corresponding to a plurality of key point pairs may be determined based on the first original image and the second original image corresponding to angle A. When the angle of the mechanical galvanometer is angle B, three-dimensional points corresponding to a plurality of key point pairs may be determined based on the first original image and the second original image corresponding to angle B. In this way, a three-dimensional reconstructed image can be generated based on three-dimensional points corresponding to all angles, that is, a complete three-dimensional reconstructed image of the surface of the object to be measured can be obtained.

[0043] In one possible embodiment, the three-dimensional imaging device may further include a third camera, i.e., an auxiliary camera. On this basis, when the multi-line laser projects N line structured lights onto the object to be measured, a third original image (e.g., line structured light image A3, line structured light image B3, etc.) of the object to be measured collected by the third camera may be obtained, and a third target image corresponding to the third original image may be determined. The third target image includes N third light-emitting areas corresponding to the N line structured lights. A third light strip centerline corresponding to each third light-emitting area in the third target image is determined. For each line structured light, a target third light strip centerline corresponding to the line structured light is determined from all the third light strip centerlines. For each first pixel point on the target first light strip centerline, projecting the first pixel point onto the second target image to obtain a projected pixel point corresponding to the first pixel point may further include determining a third pixel point from a target third light strip centerline having the same pixel height as the first pixel point, determining a target 3D reconstruction point based on the first pixel point, the third pixel point and camera calibration parameters, and transforming the target 3D reconstruction point into the projected pixel point based on a third calibration equation, where the third calibration equation represents a functional relationship between the pixel points in the second target image and the 3D reconstruction point.

[0044] In the embodiment of the present invention, the multi-line laser projects N line structured lights onto the object each time, where N is a positive integer greater than 1, such as 7, 11, 15, etc., so that the line structured light image collected by the camera each time includes N light strip center lines. The line structured light image is equivalent to the line structured light images of N positions on the object to be measured. This can reduce the number of collections of the line structured light images and reduce the time for three-dimensional reconstruction. When scanning the surface of the object to be measured with the multi-line laser, the entire contour data of the object to be measured can be quickly obtained, and the three-dimensional image information of the object to be measured can be output, improving the detection accuracy and detection speed. In addition, the first camera and the second camera are used to simultaneously collect line structured light images, and based on the line structured light images collected by the two cameras, three-dimensional information of the object to be measured can be obtained by triangulation, that is, depth information of the object to be measured can be obtained, so that the depth information of the multi-line laser light can be obtained by a single collected image, the efficiency of a single scan can be improved by N times, and a full-size scan of the entire contour of the object to be measured can be quickly realized.

[0045] The above technical solutions of the embodiments of the present invention will be described below with reference to specific application scenarios.

[0046] Application scenario 1: The three-dimensional imaging device may include a first camera, a second camera, a processor, a multi-line laser, and a mechanical galvanometer. The first camera is a left camera and the second camera is a right camera, or the first camera is a right camera and the second camera is a left camera. In application scenario 1, the camera calibration parameters, the first calibration equation, and the second calibration equation corresponding to the three-dimensional imaging device may be obtained in advance, and the camera calibration parameters, the first calibration equation, and the second calibration equation may be stored in the three-dimensional imaging device.

[0047] Exemplarily, the camera calibration parameters may include camera internal parameters of the first camera, camera internal parameters of the second camera, and camera external parameters between the first camera and the second camera. The camera internal parameters of the first camera are parameters related to the characteristics of the first camera itself, such as focal length, pixel size, distortion coefficient, etc. The camera internal parameters of the second camera are parameters related to the characteristics of the second camera itself, such as focal length, pixel size, distortion coefficient, etc. The camera external parameters between the first camera and the second camera are parameters in a world coordinate system, such as the position and rotation direction of the first camera, the position and rotation direction of the second camera, and the positional relationship between the first camera and the second camera, such as a rotation matrix and a translation matrix.

[0048] The intrinsic camera parameters of the first camera are parameters inherent to the first camera. The intrinsic camera parameters of the first camera are already given when the first camera is shipped. The intrinsic camera parameters of the second camera are parameters inherent to the second camera. The intrinsic camera parameters of the second camera are already given when the second camera is shipped.

[0049] For camera extrinsic parameters between a first camera and a second camera, such as a rotation matrix and a translation matrix, a plurality of calibration points are arranged in a target scene, a first calibration image of the target scene is collected by a first camera, the first calibration image includes the plurality of calibration points, and a second calibration image of the target scene is collected by a second camera, the second calibration image includes the plurality of calibration points. The camera extrinsic parameters between the first camera and the second camera can be determined based on pixel coordinates of the plurality of calibration points in the first calibration image and pixel coordinates of the plurality of calibration points in the second calibration image, and the process of determining the camera extrinsic parameters is not limited.

[0050] Exemplarily, the first calibration equation represents a functional relationship between pixel points in an image (referred to as image s1) collected by a first camera and three-dimensional reconstruction points, and the second calibration equation represents a functional relationship between pixel points in an image (referred to as image s2) collected by a second camera and three-dimensional reconstruction points. Assuming that a multi-line laser projects N line structured lights onto a measurement target and the angles of a mechanical galvanometer are M kinds of angles in total, a total of N×M first calibration equations and N×M second calibration equations need to be obtained, and both the first calibration equation and the second calibration equation may be light plane equations. The method of obtaining the first calibration equation and the second calibration equation may include steps S11 to S15.

[0051] In step S11, when the multi-line laser projects N line structured lights onto a white background plate for each angle of the mechanical galvanometer, an image s1 collected by a first camera is obtained, and an image s2 collected by a second camera is obtained, where the image s1 includes N first light-emitting areas corresponding to the N line structured lights, and the N first light-emitting areas are in one-to-one correspondence with the N line structured lights. The image s2 includes N second light-emitting areas corresponding to the N line structured lights, and the N second light-emitting areas are in one-to-one correspondence with the N line structured lights.

[0052] In step S12, a first light strip centerline corresponding to each first light-emitting area in the image s1 is determined, and a second light strip centerline corresponding to each second light-emitting area in the image s2 is determined, that is, N first light strip centerlines corresponding to the N line structured lights are obtained, and N second light strip centerlines corresponding to the N line structured lights are obtained.

[0053] In step S13, a plurality of keypoint pairs are determined based on all the first light strip centerlines and all the second light strip centerlines, where each keypoint pair includes a first center point on the first light strip centerline and a second center point on the second light strip centerline, and the first center point and the second center point are pixel points corresponding to the same position point on the white background board.

[0054] For example, assume that the N line structured lights are line structured light 1 and line structured light 2, the image s1 includes a first light-emitting area 1 corresponding to the line structured light 1 and a first light-emitting area 2 corresponding to the line structured light 2, and the image s2 includes a second light-emitting area 1 corresponding to the line structured light 1 and a second light-emitting area 2 corresponding to the line structured light 2. The first light-emitting area 1 corresponds to the first light strip centerline 1, the first light-emitting area 2 corresponds to the first light strip centerline 2, the second light-emitting area 1 corresponds to the second light strip centerline 1, and the second light-emitting area 2 corresponds to the second light strip centerline 2.

[0055] For example, when the object to be measured is a white background plate and two line structured lights are projected onto the white background plate, the light emitting area is relatively clear and no noise is generated, so when determining the first light strip centerline 1 based on the first light emitting area 1, each row of the first light strip centerline 1 only has one center point, and similarly, each row of the second light strip centerline 1 only has one center point. Based on this, the center point of the first row of the first light strip centerline 1 and the center point of the first row of the second light strip centerline 1 are configured as a key point pair 11, and the center point of the second row of the first light strip centerline 1 and the center point of the second row of the second light strip centerline 1 are configured as a key point pair 12, and thus inferred. Similarly, the center point of the first row of the first light strip centerline 2 and the center point of the first row of the second light strip centerline 2 are configured as key point pair 21, and the center point of the second row of the first light strip centerline 2 and the center point of the second row of the second light strip centerline 2 are configured as key point pair 22, by analogy.

[0056] In step S14, for each keypoint pair, a 3D point corresponding to the keypoint pair is determined based on the keypoint pair and the camera calibration parameters. For example, a triangulation method may be used to determine a 3D point corresponding to the keypoint pair. The triangulation method can be referred to in the following embodiments, and the description thereof will be omitted here.

[0057] In step S15, a first calibration equation and a second calibration equation corresponding to the angle of the mechanical galvanometer and the line structured light are determined based on the key point pair and the 3D point corresponding to the key point pair.

[0058] For example, for an angle A of a mechanical galvanometer, a first calibration equation and a second calibration equation corresponding to the angle A and the line structured light 1 are determined based on a plurality of key point pairs (e.g., key point pair 11, key point pair 12, etc.) between the first light strip center line 1 and the second light strip center line 1, and three-dimensional points corresponding to each key point pair.

[0059] For example, a first calibration equation may be determined based on a large number of center points of the first light strip centerline 1 and the three-dimensional points corresponding to each center point. The first calibration equation is used to express the functional relationship between the pixel points in the image s1 (i.e., the center points of the first light strip centerline 1) and the three-dimensional reconstruction points (i.e., the three-dimensional points corresponding to the center points). For example, fitting is performed using a planar model or a quadratic model to obtain the first calibration equation.

[0060] A second calibration equation may be determined based on a large number of center points of the second light strip centerline 1 and the three-dimensional points corresponding to each center point. The second calibration equation is used to express a functional relationship between pixel points in image s2 (i.e., center points of the second light strip centerline 1) and the three-dimensional reconstruction points (i.e., three-dimensional points corresponding to the center points).

[0061] Similarly, a first calibration equation and a second calibration equation corresponding to angle A and line structured light 2 can be obtained, and a first calibration equation and a second calibration equation corresponding to angle B and line structured light 1 can be obtained, by analogy.

[0062] From the above, it is possible to obtain the first calibration equation and the second calibration equation corresponding to each line structured light for each angle of the mechanical galvanometer, i.e., N×M first calibration equations and N×M second calibration equations.

[0063] Exemplarily, for N line structured lights projected by a multi-line laser, each line structured light may be numbered according to an actual order, for example, N line structured lights (i.e., laser beams) are sequentially represented as 1, 2, 3, ..., N according to the order from left to right (or from right to left), which facilitates matching and indexing of each line structured light.

[0064] In the above application scenario 1, as shown in FIG. 3, the image reconstruction method of this embodiment may include steps 301 to 309.

[0065] In step 301, when a multi-line laser projects N line structured light onto an object to be measured, a first original image of the object to be measured is acquired by a first camera, and a second original image of the object to be measured is acquired by a second camera. The acquisition time of the first original image and the acquisition time of the second original image may be the same.

[0066] Exemplarily, the first original image includes N first light-emitting regions corresponding to N line structured lights, for example, a first light-emitting region 1 corresponding to the line structured light 1, a first light-emitting region 2 corresponding to the line structured light 2, ..., etc. The second original image includes N second light-emitting regions corresponding to N line structured lights, for example, a second light-emitting region 1 corresponding to the line structured light 1, a second light-emitting region 2 corresponding to the line structured light 2, ..., etc.

[0067] In step 302, binocular correction is performed on the first original image and the second original image to obtain a first target image corresponding to the first original image and a second target image corresponding to the second original image.

[0068] For example, binocular correction is used to make the same position point on the object to be measured have the same pixel height in the first target image and the second target image. That is, the first original image and the second original image are corrected to the same pixel height for the same position point on the object to be measured by binocular correction, so that when matching, matching is performed directly within one row, which makes matching more convenient. For example, since matching corresponding points in a two-dimensional space takes a very long time, in order to reduce the search range for matching, the matching of corresponding points can be reduced from a two-dimensional search to a one-dimensional search by using an epipolar constraint. The function of binocular correction is to make the first original image and the second original image correspond to each other in a row unit to obtain the first target image and the second target image. This makes the epipolar lines of the first target image and the second target image exactly the same horizontal line, and ensures that any point on the first target image and the corresponding point on the second target image have the same row number, and one-dimensional search can be performed in that row.

[0069] Exemplarily, the first target image includes N first light-emitting regions corresponding to N line structured light, for example, a first light-emitting region 1 corresponding to the line structured light 1, a first light-emitting region 2 corresponding to the line structured light 2, ..., etc. The second target image includes N second light-emitting regions corresponding to N line structured light, for example, a second light-emitting region 1 corresponding to the line structured light 1, a second light-emitting region 2 corresponding to the line structured light 2, ..., etc.

[0070] In step 303, a first light strip centerline corresponding to each first light emitting area in the first target image is determined, and a second light strip centerline corresponding to each second light emitting area in the second target image is determined.

[0071] Exemplarily, each row of the first light-emitting regions may include a plurality of pixel points, and a center point of the row may be selected from the plurality of pixel points. The center points of all the rows of the first light-emitting regions constitute a first light strip centerline, thereby obtaining a first light strip centerline 1 corresponding to the first light-emitting region 1, a first light strip centerline 2 corresponding to the first light-emitting region 2, ..., etc. Similarly, a second light strip centerline 1 corresponding to the second light-emitting region 1, a second light strip centerline 2 corresponding to the second light-emitting region 2, ..., etc.

[0072] For example, a light strip centerline extraction algorithm may be used to determine the light strip centerline corresponding to the light emitting area. For example, the light strip centerline may be obtained by extracting the center point of each row of the light emitting area using methods such as Gaussian fitting, COG (Center of Gravity) or STEGER, and the present embodiment is not limited thereto.

[0073] For example, if the height of the first target image and the second target image is H, each of the first light strip centerlines includes the center point of the H row, and each of the second light strip centerlines includes the center point of the H row.

[0074] In step 304, for each line structured light, a target first light strip centerline and a target second light strip centerline corresponding to the line structured light are determined from all the first light strip centerlines and all the second light strip centerlines.

[0075] For example, a first light strip centerline 1 and a second light strip centerline 1 corresponding to a line structured light 1 are determined, and a first light strip centerline 2 and a second light strip centerline 2 corresponding to a line structured light 2 are determined, and an analogy is made thereby.

[0076] In step 305, for each line structured light, based on a first calibration equation corresponding to the line structured light and a target first light strip centerline corresponding to the line structured light, for each first pixel point on the target first light strip centerline, the first pixel point is converted to a target three-dimensional reconstruction point based on the first calibration equation.

[0077] Exemplarily, an angle of the mechanical galvanometer may be determined, i.e., an angle at which the first original image and the second original image are collected may be determined. For each line structured light, a first calibration equation corresponding to the angle and the line structured light may be selected from the N×M first calibration equations. The first calibration equation represents a functional relationship between pixel points in the first target image and three-dimensional reconstruction points, so that each first pixel point on the target first light strip centerline can be transformed into a target three-dimensional reconstruction point based on the first calibration equation.

[0078] For example, for a first light strip centerline 1 corresponding to the line structured light 1, each first pixel point on the first light strip centerline 1 is converted into a target three-dimensional reconstruction point based on a first calibration equation corresponding to the line structured light 1. For a first light strip centerline 2 corresponding to the line structured light 2, each first pixel point on the first light strip centerline 2 is converted into a target three-dimensional reconstruction point based on a first calibration equation corresponding to the line structured light 2, thereby making an analogy.

[0079] In step 306, for each target 3D reconstruction point corresponding to the first pixel point, the target 3D reconstruction point may be transformed into a projected pixel point in a second target image based on a second calibration equation corresponding to the line structured light, the projected pixel point being a projected pixel point corresponding to the first pixel point.

[0080] For example, for each line structured light, select a second calibration equation corresponding to the line structured light from the N×M second calibration equations, where the second calibration equation represents a functional relationship between a pixel point in the second target image and a three-dimensional reconstruction point, so that after the first pixel point on the target first light strip center line is transformed into a target three-dimensional reconstruction point, the target three-dimensional reconstruction point can be transformed into a projected pixel point based on the second calibration equation.

[0081] For example, when converting the target 3D reconstruction points into projected pixel points based on the second calibration equation corresponding to the line structured light 1, projected pixel points corresponding to each first pixel point on the first light strip centerline 1 are obtained. Also, when converting the target 3D reconstruction points into projected pixel points based on the second calibration equation corresponding to the line structured light 2, projected pixel points corresponding to each first pixel point on the first light strip centerline 2 are obtained, and so on.

[0082] In summary, for each first pixel point on the target first light strip centerline, the first pixel point can be projected onto the second target image to obtain a projected pixel point corresponding to the first pixel point.

[0083] In step 307, for each first pixel point, after obtaining the projected pixel point corresponding to the first pixel point, select the second pixel point corresponding to the projected pixel point from the target second light strip centerline.

[0084] For example, for each line structured light, taking the line structured light 1 as an example, a target second light strip centerline corresponding to the line structured light 1 may be determined, i.e., the second light strip centerline 1. For each first pixel point on the first light strip centerline 1 corresponding to the line structured light 1, after obtaining a projected pixel point corresponding to the first pixel point, a second pixel point corresponding to the projected pixel point may be selected from the second light strip centerline 1.

[0085] Obviously, the first pixel point and the second pixel point may constitute a key point pair, that is, the key point pair includes a first pixel point on the first light strip center line 1 and a second pixel point on the second light strip center line 1, and the first pixel point and the second pixel point are pixel points corresponding to the same position point on the object to be measured. The first pixel point is a pixel point in a first target image, and the second pixel point is a pixel point in a second target image.

[0086] In one possible embodiment, selecting a second pixel point corresponding to the projected pixel point from the target second light strip centerline may include determining a pixel point having the same pixel height as the projected pixel point from the target second light strip centerline, and if the determined pixel point is one, selecting the pixel point as the second pixel point, and if the determined pixel points are at least two, determining a reprojection error between the at least two pixel points and the projected pixel point, and selecting the pixel point corresponding to the smallest reprojection error as the second pixel point.

[0087] For example, a row of the second light strip centerline may include one pixel point, and in this case, if there is one pixel point with the same pixel height as the projected pixel point, the pixel point is selected as the second pixel point. Also, a row of the second light strip centerline may include at least two pixel points, and for example, if there is noise in the light-emitting area, there will be at least two pixel points in a row, and in this case, if there are at least two pixel points with the same pixel height as the projected pixel point, the reprojection error between the projected pixel point and each pixel point is determined. This determination manner is not particularly limited. After obtaining the reprojection error between the projected pixel point and each pixel point, the pixel point corresponding to the minimum reprojection error may be selected as the second pixel point.

[0088] In step 308, a 3D point corresponding to the keypoint pair is determined based on the keypoint pair and the camera calibration parameters.

[0089] For example, for each keypoint pair, the keypoint pair includes a first pixel point in a first target image and a second pixel point in a second target image, and the first pixel point and the second pixel point are pixel points corresponding to the same position point on the object to be measured. Based on this, a 3D point corresponding to the keypoint pair can be determined by triangulation, and the process will be described below by combining specific steps.

[0090] In step 3081, distortion correction is performed on a first pixel point using the camera internal parameters of a first camera, and the pixel point after distortion correction is transformed into first homogeneous coordinates. Distortion correction is performed on a second pixel point using the camera internal parameters of a second camera, and the pixel point after distortion correction is transformed into second homogeneous coordinates.

[0091] For example, due to factors such as the manufacturing accuracy of the lens and the variation in the assembly process, distortions such as radial distortion and tangential distortion exist in the image collected by the first camera. To solve the distortion problem, the camera internal parameters of the first camera include distortion parameters such as radial distortion parameters k1, k2, k3 and tangential distortion parameters p1, p2. Based on this, in this embodiment, the camera internal parameters of the first camera may be used to perform distortion correction on the first pixel point to obtain pixel coordinates after the distortion removal process. After obtaining the pixel coordinates after the distortion removal process, the pixel coordinates after the distortion removal process may be converted into first homogeneous coordinates. Similarly, the camera internal parameters of the second camera may be used to perform distortion correction on the second pixel coordinates to obtain pixel coordinates after the distortion correction, and the pixel coordinates after the distortion correction may be converted into second homogeneous coordinates. In summary, the homogeneous coordinates of the keypoint pair may include a first homogeneous coordinate of the first keypoint and a second homogeneous coordinate of the second keypoint.

[0092] In step 3082, a three-dimensional point corresponding to the keypoint pair is determined using a triangulation method based on the first homogeneous coordinates, the second homogeneous coordinates, the camera internal parameters of the first camera, the camera internal parameters of the second camera, and the camera external parameters (e.g., positional relationship, etc.) between the first camera and the second camera.

[0093] For example, as shown in Figure 4, which is a schematic diagram of the principle of the triangulation method, L is the position of the first camera, and O R is the position of the second camera, and based on the camera extrinsic parameters between the first and second cameras, O L and O R For a three-dimensional point P in three-dimensional space, the imaging position on the image plane of the first camera is p l and the imaging position on the image plane of the second camera is p r It is. l Let p be the first pixel point. r Let O be the second pixel point, and the first pixel point and the second pixel point form a keypoint pair. The 3D point P is the 3D point corresponding to the keypoint pair.L , O R , p l and p r Transform to the same coordinate system and O L , O R , p l and p r About O L andp l There is a line a1 between R andp r There is a straight line a2 between the line a1 and the line a2. If an intersection exists between the line a1 and the line a2, the intersection between the line a1 and the line a2 is a three-dimensional point P. If an intersection does not exist between the line a1 and the line a2, the three-dimensional point P is the point closest to the line a1 and the line a2. Based on the above application scenario, a three-dimensional point corresponding to the key point pair can be obtained by obtaining the three-dimensional space coordinates of the three-dimensional point P through a triangulation method. Of course, the above embodiment is merely an example of a triangulation method, and the embodiment of the triangulation method is not limited.

[0094] In summary, for each keypoint pair, a 3D point corresponding to the keypoint pair can be obtained, and since the first target image includes N first light strip centerlines, and each first light strip centerline includes H first pixel points, N×H keypoint pairs can be obtained, and the N×H keypoint pairs correspond to N×H 3D points.

[0095] In step 309, a 3D reconstruction image is generated based on the 3D points corresponding to the plurality of keypoint pairs.

[0096] For example, for each angle of the mechanical galvanometer, N×H three-dimensional points at that angle may be determined using steps 301 to 308. In the scanning process of the mechanical galvanometer, a set of original images may be obtained at each angle to perform the above operation. Assuming that the mechanical galvanometer has a total of M angles, M×N×H three-dimensional points at M angles are obtained. Based on this, a three-dimensional reconstruction image, i.e., point cloud data, may be generated based on the M×N×H three-dimensional points, and the three-dimensional reconstruction image may be output. Alternatively, the three-dimensional reconstruction image may be projected onto a camera to obtain a depth image, and the depth image may be output.

[0097] Application scenario 2: The three-dimensional imaging device may include a first camera, a second camera, a third camera, a processor, a multi-line laser, and a mechanical galvanometer. The first camera is a left camera, the second camera is a right camera, and the third camera is an auxiliary camera, or the first camera is a right camera, the second camera is a left camera, and the third camera is an auxiliary camera. The camera calibration parameters and the third calibration equation corresponding to the three-dimensional imaging device may be obtained in advance, and the camera calibration parameters and the third calibration equation may be stored in the three-dimensional imaging device.

[0098] Exemplarily, the camera calibration parameters include camera internal parameters of a first camera, camera internal parameters of a second camera, camera internal parameters of a third camera, camera external parameters between the first camera and the second camera (e.g., positional relationships such as rotation matrices and translation matrices), camera external parameters between the first camera and the third camera (e.g., positional relationships), and camera external parameters between the second camera and the third camera (e.g., positional relationships).

[0099] For the method of obtaining the camera calibration parameters, refer to Application Scenario 1, and the explanation will be omitted here.

[0100] For example, the third calibration equation represents a functional relationship between pixel points in an image (denoted as image s2) collected by the second camera and three-dimensional reconstruction points. Assuming that the multi-line laser projects N line structured lights onto the object to be measured and the mechanical galvanometer has a total of M angles, a total of N×M third calibration equations need to be obtained, and each of the third calibration equations may be a light plane equation. The method of obtaining the third calibration equation can be referred to the method of obtaining the second calibration equation in application scenario 1, and the description is omitted here.

[0101] Exemplarily, for N line structured lights projected by a multi-line laser, each line structured light may be numbered according to an actual order, for example, N line structured lights (i.e., laser beams) are sequentially represented as 1, 2, 3, ..., N according to the order from left to right (or from right to left), which facilitates matching and indexing of each line structured light.

[0102] In the above application scenario 2, as shown in FIG. 5, the image reconstruction method of this embodiment may include steps 501 to 509.

[0103] In step 501, when a multi-line laser projects N lines of structured light onto an object to be measured, a first original image of the object to be measured is acquired by a first camera, a second original image of the object to be measured is acquired by a second camera, and a third original image of the object to be measured is acquired by a third camera. The acquisition time of the first original image, the acquisition time of the second original image, and the acquisition time of the third original image may be the same.

[0104] For example, the first original image may include N first light-emitting regions corresponding to N line structured lights, the second original image may include N second light-emitting regions corresponding to N line structured lights, and the third original image may include N third light-emitting regions corresponding to N line structured lights.

[0105] In step 502, trinocular correction is performed on the first original image, the second original image, and the third original image to obtain a first target image corresponding to the first original image, a second target image corresponding to the second original image, and a third target image corresponding to the third original image. Here, trinocular correction is used to make the same position point on the object to be measured have the same pixel height in the first target image, the second target image, and the third target image. That is, for the same position point on the object to be measured, the first original image, the second original image, and the third original image can be corrected to the same pixel height by trinocular correction.

[0106] For example, the first target image may include N first light-emitting areas corresponding to N line structured lights, the second target image may include N second light-emitting areas corresponding to N line structured lights, and the third target image may include N third light-emitting areas corresponding to N line structured lights.

[0107] In step 503, a first light strip centerline corresponding to each first light emitting area in the first target image is determined, a second light strip centerline corresponding to each second light emitting area in the second target image is determined, and a third light strip centerline corresponding to each third light emitting area in the third target image is determined.

[0108] For example, a light strip centerline extraction algorithm may be used to determine the light strip centerline corresponding to the light emitting area. For example, Gaussian fitting, COG, STEGER or other methods may be used to extract the center point of each row of the light emitting area to obtain the light strip centerline, and this embodiment is not limited thereto.

[0109] In step 504, for each line structured light, a target first light strip centerline corresponding to the line structured light is determined from all of the first light strip centerlines, a target second light strip centerline corresponding to the line structured light is determined from all of the second light strip centerlines, and a target third light strip centerline corresponding to the line structured light is determined from all of the third light strip centerlines.

[0110] In step 505, based on the target first light strip centerline and the target third light strip centerline corresponding to the line structured light, for each first pixel point on the target first light strip centerline, a third pixel point having the same pixel height from the target third light strip centerline as the first pixel point is determined, and a target three-dimensional reconstruction point corresponding to the first pixel point is determined based on the first pixel point, the third pixel point and camera calibration parameters.

[0111] Exemplarily, for each first pixel point on the target first light strip centerline, a pixel point having the same pixel height as the first pixel point from the target third light strip centerline is determined. If the determined pixel point is one, the pixel point is selected as a third pixel point. If the determined pixel points are at least two, the reprojection error between the at least two pixel points and the first pixel point is determined, and the pixel point corresponding to the minimum reprojection error is selected as a third pixel point. The manner of determining the reprojection error is not limited.

[0112] Exemplarily, the first pixel point and the third pixel point may constitute one keypoint pair. That is, the keypoint pair includes a first pixel point in a first target image and a third pixel point in a third target image, and the first pixel point and the third pixel point are pixel points corresponding to the same position point on the object to be measured. Based on this, a three-dimensional point corresponding to the keypoint pair can be determined by a triangulation method, and the three-dimensional point is a target three-dimensional reconstruction point corresponding to the first pixel point. For example, the first pixel point is corrected for distortion according to the camera internal parameters of the first camera, the pixel point after distortion correction is transformed into a first homogeneous coordinate, and the third pixel point is corrected for distortion according to the camera internal parameters of the third camera, and the pixel point after distortion correction is transformed into a third homogeneous coordinate. A three-dimensional point corresponding to the keypoint pair is determined by a triangulation method based on the first homogeneous coordinates, the third homogeneous coordinates, the camera internal parameters of the first camera, the camera internal parameters of the third camera, and the camera external parameters (e.g., positional relationship, etc.) between the first camera and the third camera.

[0113] In summary, for each first pixel point on the target first light strip centerline, a target 3D reconstruction point corresponding to the first pixel point is determined, and a correspondence relationship between the first pixel point and the target 3D reconstruction point is obtained.

[0114] In step 506, for each target 3D reconstruction point corresponding to the first pixel point, the target 3D reconstruction point may be transformed into a projected pixel point in a second target image based on a third calibration equation corresponding to the line structured light, the projected pixel point being a projected pixel point corresponding to the first pixel point.

[0115] For example, for each line structured light, select a third calibration equation corresponding to the line structured light from the N×M third calibration equations, where the third calibration equation represents a functional relationship between a pixel point in the second target image and a three-dimensional reconstruction point, so that after the first pixel point on the target first light strip center line is transformed into a target three-dimensional reconstruction point, the target three-dimensional reconstruction point can be transformed into a projected pixel point according to the third calibration equation.

[0116] In summary, for each first pixel point on the target first light strip centerline, the first pixel point can be projected onto the second target image to obtain a projected pixel point corresponding to the first pixel point.

[0117] In step 507, for each first pixel point, after obtaining the projected pixel point corresponding to the first pixel point, select the second pixel point corresponding to the projected pixel point from the target second light strip centerline.

[0118] For example, a pixel point having the same pixel height as the projected pixel point from the center line of the target second light strip may be determined, and if there is one determined pixel point, the pixel point may be selected as the second pixel point; if there are at least two determined pixel points, a reprojection error between the at least two pixel points and the projected pixel point may be determined, and the pixel point corresponding to the smallest reprojection error may be selected as the second pixel point.

[0119] Obviously, the first pixel point and the second pixel point may constitute a keypoint pair, the first pixel point and the second pixel point being pixel points corresponding to the same position on the object to be measured, the first pixel point being a pixel point in a first target image, and the second pixel point being a pixel point in a second target image.

[0120] In step 508, a 3D point corresponding to the keypoint pair is determined based on the keypoint pair and the camera calibration parameters. For example, a first pixel point is corrected for distortion according to the internal camera parameters of the first camera, and the corrected pixel point is transformed into a first homogeneous coordinate, and a second pixel point is corrected for distortion according to the internal camera parameters of the second camera, and the corrected pixel point is transformed into a second homogeneous coordinate. A 3D point corresponding to the keypoint pair is determined by triangulation according to the first homogeneous coordinate, the second homogeneous coordinate, the internal camera parameters of the first camera, the internal camera parameters of the second camera, and the external camera parameters between the first camera and the second camera.

[0121] In summary, for each keypoint pair, we can obtain the 3D point that corresponds to that keypoint pair.

[0122] In step 509, a 3D reconstruction is generated based on the 3D points corresponding to the plurality of keypoint pairs.

[0123] In the embodiment of the present invention, the multi-line laser projects N lines of line structured light onto the object to be measured each time, so that the line structured light image collected by the camera each time includes N light strip center lines. The line structured light image is equivalent to the line structured light image of N positions on the object to be measured. This can reduce the number of collections of the line structured light image and reduce the time of three-dimensional reconstruction. When scanning the surface of the object to be measured with the multi-line laser, the entire contour data of the object to be measured can be quickly acquired, and the three-dimensional image information of the object to be measured can be output, so as to improve the detection accuracy and detection speed. In addition, the first camera and the second camera are used to simultaneously collect the line structured light images, and the three-dimensional information of the object to be measured can be obtained by triangulation based on the line structured light images collected by the two cameras, that is, the depth information of the object to be measured can be obtained, so that the depth information in the multi-line laser light can be obtained by one collected image, and the efficiency of one scan can be improved by N times, and the full-size scan of the entire contour of the object to be measured can be quickly realized. By using multi-line laser triangulation to obtain depth information of multiple laser beams at one time, and using a mechanical galvanometer to realize scanning with small angle between laser beams, it is possible to complete high-precision scanning of the entire surface contour of the object to be measured. The laser has high contrast, good collimation, a relatively large depth of field, better adaptability to materials in three-dimensional detection applications, and higher detection accuracy, so it can be applied to three-dimensional measurement applications in the field of machine vision and three-dimensional grasping and positioning applications in the field of industrial automation.

[0124] Based on the same idea as the above method, an embodiment of the present invention provides an image reconstruction device applied to a three-dimensional imaging device, the three-dimensional imaging device includes a first camera, a second camera and a multi-line laser. As shown in Figure 6, a schematic block diagram of the device can include an acquisition module 61, a determination module 62 and a generation module 63.

[0125] The acquisition module 61 is configured to acquire a first original image of the object collected by the first camera and acquire a second original image of the object collected by the second camera when the multi-line laser projects N lines of structured light onto the object, where N is a positive integer greater than 1.

[0126] The determining module 62 is configured to determine a first target image corresponding to a first original image and a second target image corresponding to a second original image, determine a first light strip centerline corresponding to each first light-emitting area in the first target image, determine a second light strip centerline corresponding to each second light-emitting area in the second target image, determine a plurality of keypoint pairs based on all the first light strip centerlines and all the second light strip centerlines, and determine three-dimensional points corresponding to the keypoint pairs based on the keypoint pairs and camera calibration parameters, where the first target image includes N first light-emitting areas corresponding to the N line structured lights, the second target image includes N second light-emitting areas corresponding to the N line structured lights, and for each keypoint pair, the keypoint pair includes a first pixel point on the first light strip centerline and a second pixel point on the second light strip centerline, and the first pixel point and the second pixel point are pixel points corresponding to the same position point on the measured object.

[0127] The generation module 63 is configured to generate a 3D reconstructed image of the object based on 3D points corresponding to the plurality of keypoint pairs.

[0128] Exemplarily, when determining a first target image corresponding to a first original image and a second target image corresponding to a second original image, the determining module 62 specifically: configured to determine the first original image as a first target image and to determine the second original image as a second target image; or A binocular correction is performed on a first original image and a second original image to obtain a first target image corresponding to the first original image and a second target image corresponding to the second original image; The binocular correction is used to ensure that the same location point on the object has the same pixel height in the first target image and the second target image.

[0129] Illustratively, when determining a plurality of key point pairs based on all the first light strip centerlines and all the second light strip centerlines, the determining module 62 specifically includes: For each line structured light, determine a target first light strip centerline and a target second light strip centerline corresponding to the line structured light from all the first light strip centerlines and all the second light strip centerlines; For each first pixel point on the target first light strip centerline, project the first pixel point onto the second target image to obtain a projected pixel point corresponding to the first pixel point, and select a second pixel point from the target second light strip centerline corresponding to the projected pixel point; A keypoint pair is configured to be generated based on the first pixel point and the second pixel point.

[0130] Illustratively, when the determining module 62 selects a second pixel point corresponding to the projected pixel point from the target second light strip center line, the determining module 62 specifically: determining a pixel point from the target second light strip centerline that has the same pixel height as the projected pixel point; If the determined pixel point is one, select the pixel point as the second pixel point; If the determined pixel points are at least two, the method is configured to determine a reprojection error between the at least two pixel points and the projected pixel point, and to select the pixel point corresponding to the smallest reprojection error as the second pixel point.

[0131] Exemplarily, when the determination module 62 projects the first pixel point onto the second target image and obtains a projected pixel point corresponding to the first pixel point, the determination module 62 specifically includes: Obtain a first calibration equation and a second calibration equation corresponding to the line structured light; Transforming the first pixel point into a target 3D reconstruction point based on the first calibration equation; configured to transform the target 3D reconstruction points to the projected pixel points based on the second calibration equation; The first calibration equation represents a functional relationship between pixel points in a first target image and three-dimensional reconstruction points, and the second calibration equation represents a functional relationship between pixel points in a second target image and three-dimensional reconstruction points.

[0132] Exemplarily, the three-dimensional imaging device further includes a third camera; The acquisition module 61 is further configured to acquire a third original image of the object to be measured, which is collected by a third camera, when a multi-line laser projects N line structured light beams onto the object to be measured; The determination module 62 is further configured to determine a third target image corresponding to the third original image; determine third light strip centerlines corresponding to each third light-emitting area in the third target image; and for each line structured light, determine a target third light strip centerline corresponding to the line structured light from all the third light strip centerlines, where the third target image includes N third light-emitting areas corresponding to the N line structured lights; For each first pixel point on the target first light strip center line, the determining module 62 projects the first pixel point onto the second target image to obtain a projected pixel point corresponding to the first pixel point, specifically: determining a third pixel point having the same pixel height from the target third light strip centerline as the first pixel point; determining a target 3D reconstruction point based on the first pixel point, the third pixel point and camera calibration parameters; configured to transform the target 3D reconstruction points to the projected pixel points based on a third calibration equation; A third calibration equation represents a functional relationship between pixel points in the second target image and the 3D reconstructed points.

[0133] Exemplarily, the camera calibration parameters include camera intrinsic parameters of a first camera, camera intrinsic parameters of a second camera, and camera extrinsic parameters between the first camera and the second camera; When determining the 3D points corresponding to the keypoint pairs based on the keypoint pairs and the camera calibration parameters, the determination module 62 specifically: performing distortion correction on the first pixel point using internal camera parameters of a first camera, and converting the pixel point after distortion correction into first homogeneous coordinates; performing distortion correction on the second pixel point using internal camera parameters of a second camera, and converting the pixel point after distortion correction into second homogeneous coordinates; The method is configured to determine a three-dimensional point corresponding to the key point pair in a triangulation manner based on the first homogeneous coordinates, the second homogeneous coordinates, the intrinsic camera parameters of the first camera, the intrinsic camera parameters of the second camera, and the extrinsic camera parameters.

[0134] Based on a similar idea to the above method, an embodiment of the present invention provides a three-dimensional imaging device including a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions executable by the processor, and the processor is configured to execute the image reconstruction method disclosed in the above embodiment of the present invention by executing the machine-executable instructions.

[0135] Based on a similar idea to the above method, an embodiment of the present invention further provides a machine-readable storage medium, in which several computer instructions are stored, and when the computer instructions are executed by a processor, the processor performs the image reconstruction method disclosed in the above embodiment of the present invention.

[0136] The machine-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, the machine-readable storage medium may be a random access memory (RAM), a volatile memory, a non-volatile memory, a flash memory, a storage drive (e.g., a hard disk drive), a solid-state drive, any type of storage disk (e.g., an optical disk, a DVD, etc.), or a similar storage medium, or a combination thereof.

[0137] The systems, devices, modules, or units described in the above embodiments may be specifically realized by an entity or a product having some function. A typical implementation device is a computer, and the specific form of the computer may be a personal computer, a laptop computer, a mobile phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an e-mail sending and receiving device, a game machine, a tablet PC, a wearable device, or any combination of these devices.

[0138] For convenience of description, the above-mentioned device will be described as being divided into units according to their functions. Of course, when implementing the present invention, the functions of each unit can be realized by the same or multiple pieces of software and / or hardware.

[0139] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, a system, or a computer program product. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product embodied in one or more computer usable storage mediums (including, but not limited to, magnetic disk memories, CD-ROMs, optical memories, etc.) having computer usable program code therein.

[0140] The present invention will be described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be realized by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor, or other programmable data processing device to generate a machine, and the instructions executed by the processor of the general purpose computer or other programmable data processing device generate an apparatus for implementing the function specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0141] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specified manner, where the instructions stored in the computer-readable memory produce an article of manufacture that includes an instruction apparatus that implements the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0142] These computer program instructions may be loaded into a computer or other programmable data processing device and a series of operational steps executed on the computer or other programmable device to generate a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0143] The above description is merely an embodiment of the present invention, and is not intended to limit the present invention. Various modifications and variations of the present invention are possible for those skilled in the art. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. 1. An image reconstruction method applied to a three-dimensional imaging device, the three-dimensional imaging device including a first camera, a second camera, and a multi-line laser, the method comprising: When the multi-line laser projects N lines of structured light onto the object, acquiring a first original image of the object collected by the first camera and acquiring a second original image of the object collected by the second camera, where N is a positive integer greater than 1; determining a first target image corresponding to the first original image and a second target image corresponding to the second original image, the first target image including N first light-emitting regions corresponding to the N line structured lights, and the second target image including N second light-emitting regions corresponding to the N line structured lights; determining a first light strip centerline corresponding to each first light emitting area in the first target image and determining a second light strip centerline corresponding to each second light emitting area in the second target image; determining a plurality of keypoint pairs based on all of the first light strip centerlines and all of the second light strip centerlines, where for each keypoint pair, the keypoint pair includes a first pixel point on the first light strip centerline and a second pixel point on the second light strip centerline, the first pixel point and the second pixel point being pixel points corresponding to a same location point on the object; determining 3D points corresponding to the keypoint pairs based on the keypoint pairs and camera calibration parameters; generating a 3D reconstructed image of the object based on 3D points corresponding to the plurality of key point pairs; 2. An image reconstruction method comprising:

2. Determining a first target image corresponding to the first original image and a second target image corresponding to the second original image includes: determining the first original image as the first target image and determining the second original image as the second target image; or performing binocular correction on the first original image and the second original image to obtain a first target image corresponding to the first original image and a second target image corresponding to the second original image; The binocular correction is used to make the same position point on the object to be measured have the same pixel height in the first target image and the second target image.

2. The method of claim 1 .

3. determining a plurality of keypoint pairs based on all of the first light strip centerlines and all of the second light strip centerlines, determining, for each line structured light, a target first light strip centerline and a target second light strip centerline corresponding to the line structured light from all of the first light strip centerlines and all of the second light strip centerlines; For each first pixel point on the target first light strip centerline, project the first pixel point onto the second target image to obtain a projected pixel point corresponding to the first pixel point, and select a second pixel point from the target second light strip centerline corresponding to the projected pixel point; generating a keypoint pair based on the first pixel point and the second pixel point; 3. The method according to claim 1 or 2.

4. selecting a second pixel point from the target second light strip centerline that corresponds to the projected pixel point, determining a pixel point from the target second light strip centerline that has the same pixel height as the projected pixel point; If there is one determined pixel point, select the pixel point as the second pixel point; if the determined pixel points are at least two, determining a reprojection error between the at least two pixel points and the projected pixel point, and selecting a pixel point corresponding to a minimum reprojection error among the at least two pixel points as the second pixel point; 4. The method according to claim 3 .

5. Projecting the first pixel point onto the second target image to obtain a projected pixel point corresponding to the first pixel point includes: Obtaining a first calibration equation and a second calibration equation corresponding to the line structured light; transforming the first pixel points to target 3D reconstruction points based on the first calibration equation; and transforming the target 3D reconstruction points to the projected pixel points based on the second calibration equation; the first calibration equation represents a functional relationship between pixel points in the first target image and the three-dimensional reconstruction points, and the second calibration equation represents a functional relationship between pixel points in the second target image and the three-dimensional reconstruction points.

4. The method according to claim 3 .

6. The three-dimensional imaging device further includes a third camera, and the method further comprises: When the multi-line laser projects the N line structured light beams onto the object to be measured, a step of acquiring a third original image of the object to be measured collected by the third camera, and determining a third target image corresponding to the third original image, wherein the third target image includes N third light-emitting regions corresponding to the N line structured light beams; determining a third light strip centerline corresponding to each third light emitting area in the third target image; and determining, for each line structured light, a target third light strip centerline corresponding to the line structured light from all the third light strip centerlines; For each first pixel point on the target first light strip centerline, projecting the first pixel point onto the second target image to obtain a projected pixel point corresponding to the first pixel point includes: determining a third pixel point having the same pixel height from the target third light strip centerline as the first pixel point; determining a target 3D reconstruction point based on the first pixel point, the third pixel point and camera calibration parameters; and transforming the target 3D reconstruction points to the projected pixel points based on a third calibration equation; a third calibration equation expressing a functional relationship between pixel points in the second target image and the three-dimensional reconstruction points; 4. The method according to claim 3 .

7. the camera calibration parameters include intrinsic camera parameters of the first camera, intrinsic camera parameters of the second camera, and extrinsic camera parameters between the first camera and the second camera; Determining 3D points corresponding to the keypoint pairs based on the keypoint pairs and camera calibration parameters includes: performing distortion correction on the first pixel point using camera internal parameters of the first camera, and converting the pixel point after distortion correction into first homogeneous coordinates; performing distortion correction on the second pixel point using camera internal parameters of the second camera, and converting the pixel point after distortion correction into second homogeneous coordinates; determining a 3D point corresponding to the key point pair by triangulation based on the first homogeneous coordinates, the second homogeneous coordinates, the intrinsic camera parameters of the first camera, the intrinsic camera parameters of the second camera, and the extrinsic camera parameters; 2. The method of claim 1 .

8. An image reconstruction apparatus for application to a three-dimensional imaging apparatus, the three-dimensional imaging apparatus including a first camera, a second camera, and a multi-line laser, the apparatus comprising: an acquisition module configured to acquire a first original image of the measured object collected by the first camera and acquire a second original image of the measured object collected by the second camera when the multi-line laser projects N lines of structured light onto the measured object, where N is a positive integer greater than 1; a determination module configured to: determine a first target image corresponding to the first original image and a second target image corresponding to the second original image; determine first light strip centerlines corresponding to each first light emitting area in the first target image; determine second light strip centerlines corresponding to each second light emitting area in the second target image; determine a plurality of key point pairs based on all the first light strip centerlines and all the second light strip centerlines; and determine three-dimensional points corresponding to the key point pairs based on the key point pairs and camera calibration parameters; a generation module configured to generate a 3D reconstruction of the object based on 3D points corresponding to the plurality of key point pairs; The first target image includes N first light-emitting regions corresponding to the N line structured lights, and the second target image includes N second light-emitting regions corresponding to the N line structured lights, for each key point pair, the key point pair includes a first pixel point at the first light strip centerline and a second pixel point at the second light strip centerline, the first pixel point and the second pixel point being pixel points corresponding to the same location point on the object; 1. An image reconstruction device comprising:

9. When the determination module determines a first target image corresponding to the first original image and a second target image corresponding to the second original image, configured to determine the first original image as the first target image and to determine the second original image as the second target image; or a binocular correction is performed on the first original image and the second original image to obtain a first target image corresponding to the first original image and a second target image corresponding to the second original image; The binocular correction is used to make the same position point on the object to be measured have the same pixel height in the first target image and the second target image.

9. The apparatus of claim 8.

10. When the determination module determines a plurality of keypoint pairs based on all of the first light strip centerlines and all of the second light strip centerlines, For each line structured light, determine a target first light strip centerline and a target second light strip centerline corresponding to the line structured light from all of the first light strip centerlines and all of the second light strip centerlines; For each first pixel point on the target first light strip centerline, project the first pixel point onto the second target image to obtain a projected pixel point corresponding to the first pixel point; and select a second pixel point from the target second light strip centerline corresponding to the projected pixel point; generating a keypoint pair based on the first pixel point and the second pixel point; 10. Apparatus according to claim 8 or 9.

11. When the determination module selects a second pixel point corresponding to the projected pixel point from the target second light strip centerline, determining a pixel point from the target second light strip centerline that has the same pixel height as the projected pixel point; If there is one determined pixel point, select the pixel point as the second pixel point; if the determined pixel points are at least two, it is configured to determine a reprojection error between the at least two pixel points and the projected pixel point, and to select a pixel point corresponding to a minimum reprojection error among the at least two pixel points as the second pixel point.

11. The apparatus of claim 10.

12. When the determination module projects the first pixel point onto the second target image to obtain a projected pixel point corresponding to the first pixel point, Obtaining a first calibration equation and a second calibration equation corresponding to the line structured light; Transforming the first pixel points into target 3D reconstruction points based on the first calibration equation; configured to transform the target 3D reconstruction points to the projected pixel points based on the second calibration equation; the first calibration equation represents a functional relationship between pixel points in the first target image and the three-dimensional reconstruction points, and the second calibration equation represents a functional relationship between pixel points in the second target image and the three-dimensional reconstruction points.

11. The apparatus of claim 10.

13. the three-dimensional imaging device further includes a third camera; The acquisition module is further configured to acquire a third original image of the object collected by the third camera when the multi-line laser projects the N line structured light onto the object; The determination module is further configured to determine a third target image corresponding to the third original image; determine third light strip centerlines corresponding to each third light-emitting area in the third target image; and for each line structured light, determine a target third light strip centerline corresponding to the line structured light from all the third light strip centerlines, where the third target image includes N third light-emitting areas corresponding to the N line structured lights; For each first pixel point on the target first light strip centerline, the determination module projects the first pixel point onto the second target image to obtain a projected pixel point corresponding to the first pixel point, determining a third pixel point having the same pixel height as the first pixel point from the target third light strip centerline; determining a target 3D reconstruction point based on the first pixel point, the third pixel point and camera calibration parameters; configured to transform the target 3D reconstruction points to the projected pixel points based on a third calibration equation; a third calibration equation expressing a functional relationship between pixel points in the second target image and the three-dimensional reconstruction points; 11. The apparatus of claim 10.

14. the camera calibration parameters include intrinsic camera parameters of the first camera, intrinsic camera parameters of the second camera, and extrinsic camera parameters between the first camera and the second camera; When the determination module determines a 3D point corresponding to the key point pair based on the key point pair and camera calibration parameters, performing distortion correction on the first pixel point using camera internal parameters of the first camera, and converting the pixel point after distortion correction into first homogeneous coordinates; performing distortion correction on the second pixel point using camera internal parameters of the second camera, and converting the pixel point after distortion correction into second homogeneous coordinates; determining a three-dimensional point corresponding to the key point pair in a triangulation manner based on the first homogeneous coordinates, the second homogeneous coordinates, intrinsic camera parameters of the first camera, intrinsic camera parameters of the second camera, and the extrinsic camera parameters.

9. The apparatus of claim 8.

15. A three-dimensional imaging device including a processor and a machine-readable storage medium, The machine-readable storage medium stores machine-executable instructions executable by the processor; The processor is configured to execute the machine-executable instructions to perform the method of any one of claims 1 to 7. A three-dimensional imaging device.

Images