Image reconstruction method, device and equipment

Abstract

An image reconstruction method, device, and instrument are provided. The image reconstruction method includes the steps of: acquiring a first original image collected by a first camera and a second original image collected by a second camera when a multi-line laser projects N lines of structured light onto an object to be measured (201); determining a first target image corresponding to the first original image and a second target image corresponding to the second original image (202); and determining a first light strip centerline corresponding to each first light-emitting area in the first target image and a second light strip centerline corresponding to each second light-emitting area in the second target image. The method includes: determining (203) a plurality of keypoint pairs based on a first light strip centerline and a second light strip centerline, the keypoint pair including a first pixel point at the first light strip centerline and a second pixel point at the second light strip centerline; determining (205) three-dimensional points based on the keypoint pairs and camera calibration parameters; and generating (206) a three-dimensional reconstructed image based on the three-dimensional points corresponding to the plurality of keypoint pairs.

Description

[Technical Field]

[0001] The present invention relates to the technical field of image processing, and in particular to an image reconstruction method, device and apparatus. [Background technology]

[0002] The three-dimensional imaging device includes a laser and a camera. The laser is used to project line structured light onto the surface of the object to be measured (i.e., the object to be measured), and the camera is used to photograph the object to obtain an image having line structured light, i.e., a line structured light image. After obtaining the line structured light image, the center line of the light strip in the line structured light image is obtained, and the center line of the light strip is transformed based on predetermined sensor parameters to obtain the spatial coordinates (i.e., three-dimensional coordinates) of the object to be measured at its current position. Based on the spatial coordinates of the object to be measured at its current position, a three-dimensional reconstruction of the object to be measured can be realized. Summary of the Invention [Means for solving the problem]

[0003] An embodiment of the present invention provides an image reconstruction method applied to a three-dimensional imaging device, the three-dimensional imaging device including a first camera, a second camera, and a multi-line laser, the method comprising: When the multi-line laser projects N lines of structured light onto the object to be measured, acquiring a first original image of the object to be measured collected by the first camera and acquiring a second original image of the object to be measured collected by the second camera, where N is a positive integer greater than 1; determining a first target image corresponding to the first original image and a second target image corresponding to the second original image, wherein the first target image includes N first light-emitting regions corresponding to the N line structured lights, and the second target image includes N second light-emitting regions corresponding to the N line structured lights; determining a first light strip centerline corresponding to each first light emitting area in the first target image and determining a second light strip centerline corresponding to each second light emitting area in the second target image; determining a plurality of keypoint pairs based on all of the first light strip centerlines and all of the second light strip centerlines, where for each keypoint pair, the keypoint pair includes a first pixel point on the first light strip centerline and a second pixel point on the second light strip centerline, and the first pixel point and the second pixel point are pixel points corresponding to the same location on the object; determining a 3D point corresponding to the keypoint pair based on the keypoint pair and camera calibration parameters; and generating a three-dimensional reconstructed image of the object based on three-dimensional points corresponding to the plurality of key point pairs.

[0004] An embodiment of the present invention provides an image reconstruction apparatus for application to a three-dimensional imaging apparatus, the three-dimensional imaging apparatus including a first camera, a second camera, and a multi-line laser, the apparatus comprising: an acquisition module configured to acquire a first original image of the object to be measured collected by the first camera and acquire a second original image of the object to be measured collected by the second camera when the multi-line laser projects N lines of structured light onto the object to be measured, where N is a positive integer greater than 1; a determination module configured to: determine a first target image corresponding to the first original image and a second target image corresponding to the second original image; determine first light strip centerlines corresponding to each first light emitting area in the first target image; determine second light strip centerlines corresponding to each second light emitting area in the second target image; determine a plurality of keypoint pairs based on all the first light strip centerlines and all the second light strip centerlines; and determine three-dimensional points corresponding to the keypoint pairs based on the keypoint pairs and camera calibration parameters; a generation module configured to generate a 3D reconstructed image of the object based on 3D points corresponding to the plurality of key point pairs; the first target image includes N first light-emitting regions corresponding to the N line structured lights, and the second target image includes N second light-emitting regions corresponding to the N line structured lights; For each key point pair, the key point pair includes a first pixel point on the center line of the first light strip and a second pixel point on the center line of the second light strip, and the first pixel point and the second pixel point are pixel points corresponding to the same position point on the object to be measured.

[0005] An embodiment of the present invention provides a three-dimensional imaging device including a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions executable by the processor, and the processor is configured to execute the machine-executable instructions to perform the image reconstruction method disclosed in the above embodiment of the present invention. [Effects of the Invention]

[0006] In an embodiment of the present invention, the multi-line laser projects N line structured light beams onto the object to be measured each time, where N is a positive integer greater than 1, such as 7, 11, or 15. Therefore, the line structured light image collected by the camera each time includes N light strip center lines. The line structured light image is equivalent to line structured light images of N positions on the object to be measured. This reduces the number of times the line structured light images need to be collected and the time required for 3D reconstruction. When the multi-line laser scans the surface of the object to be measured, the entire contour data of the object can be quickly acquired, and 3D image information of the object can be output, improving detection accuracy and speed. In addition, the first camera and the second camera are used to simultaneously collect line-structured light images, and based on the line-structured light images collected by the two cameras, three-dimensional information of the object to be measured can be obtained by triangulation, that is, depth information of the object to be measured can be obtained. Thus, depth information in multi-line laser light can be obtained by a single collected image, the efficiency of a single scan can be improved by N times, and a full-size scan of the entire contour of the object to be measured can be quickly achieved. [Brief explanation of the drawings]

[0007] [Figure 1A] 1 is a schematic diagram illustrating the configuration of a three-dimensional imaging device according to an embodiment of the present invention. [Figure 1B] 1 is a schematic diagram illustrating the configuration of a three-dimensional imaging device according to an embodiment of the present invention. [Figure 1C] FIG. 1 is a schematic diagram of a multiple line laser according to an embodiment of the present invention. [Figure 1D] FIG. 1 is a schematic diagram of a multiple line laser according to an embodiment of the present invention. [Figure 2] 1 is a flowchart of an image reconstruction method according to an embodiment of the present invention. [Figure 3] 1 is a flowchart of an image reconstruction method according to an embodiment of the present invention. [Figure 4] 1 is a schematic diagram illustrating the principle of a triangulation method according to an embodiment of the present invention; [Figure 5]1 is a flowchart of an image reconstruction method according to an embodiment of the present invention. [Figure 6] 1 is a schematic configuration diagram of an image reconstruction device according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0008] The terms used in the embodiments of the present invention are not intended to limit the present invention but merely to describe specific embodiments. As used in the embodiments and claims of the present invention, the singular forms "a," "the," and "the" are also intended to include the plural form unless the context clearly indicates otherwise. As used herein, the term "and / or" should be understood to mean any and all possible combinations including one or more of the associated listed items.

[0009] In the examples herein, terms such as "first," "second," and "third" may be used to describe various pieces of information, but it should be understood that these pieces of information should not be limited to these terms. These terms are used only to distinguish between pieces of information of the same type. For example, first information may be referred to as "second information," and similarly, second information may be referred to as "first information" without departing from the scope of the present invention. Furthermore, depending on the context, the word "if" may be interpreted as "when," "when," or "in response to determining."

[0010] Related technologies for obtaining 3D reconstruction images include surface structured light projection, binocular speckle imaging, time-of-flight (TOF) imaging, and single-line laser contour scanning. Surface structured light projection employs DLP (Digital Light Processing) or LCD (Liquid Crystal Display) projection technology, and uses an LED (Light-Emitting Diode) light source as the projection light source. However, it has a large projection volume, dissipates energy, and has a large field of view at long distances, resulting in high power consumption and disadvantages for 3D positioning applications. Binocular speckle imaging combines binocular parallax and laser speckle binocular matching, resulting in low detection accuracy and poor edge contours, which are disadvantages for contour scanning and 3D positioning applications. TOF imaging is limited by the camera resolution and has a detection accuracy of centimeters, making it unsuitable for automated high-precision positioning applications. When using the single-line laser contour scanning method, the depth information of the object is scanned using a single-line laser, resulting in a slow scanning speed and poor stability, which cannot meet the positioning needs of 3D reconstruction.

[0011] Taking the single-line laser contour scanning method as an example, a laser is used to project a line structured light onto the surface of the workpiece, and a camera is used to capture an image of the workpiece, i.e., a line structured light image. After capturing the line structured light image, the spatial coordinates (i.e., three-dimensional coordinates) of the workpiece at its current position can be obtained based on the line structured light image, thereby achieving three-dimensional reconstruction of the workpiece. However, to achieve three-dimensional reconstruction of the workpiece, line structured light images must be collected at different positions on the workpiece. That is, the laser projects line structured light onto different positions on the workpiece, and each position corresponds to one line structured light image. Because the camera only collects one line structured light image for each position, multiple line structured light images must be collected to complete the three-dimensional reconstruction. That is, the three-dimensional reconstruction takes a relatively long time, has a slow scanning speed, and is unstable, which does not meet the positioning needs of three-dimensional reconstruction.

[0012] In view of this, an embodiment of the present invention provides a 3D imaging method using multi-line laser scanning, which uses triangulation to obtain depth information of an object to be measured, and a multi-line laser to form an optical scan on the surface of the object to quickly obtain the entire contour data of the object to be measured, and outputs 3D image information of the object to be measured. Here, the 3D imaging method using multi-line laser scanning can be applied in the fields of machine vision and industrial automation, and can be used to realize 3D measurement and robot positioning, and its application scenarios are not limited.

[0013] In this embodiment, the depth information of the object to be measured may be acquired using triangulation. By acquiring depth information once using a multi-line laser beam of 10 lines, 20 lines, etc., the efficiency of a single scan can be improved by 10 to 20 times. Then, by performing multi-line laser beam scanning, full-size scanning of the entire contour of the object to be measured can be achieved.

[0014] In this embodiment, the problems of large volume and high power consumption of surface structured light projection method are solved, the problems of low detection accuracy and poor edge contour of binocular speckle method are solved, the problems of low detection accuracy of TOF method are solved, and the problems of slow scanning speed and poor stability of single-line laser contour scanning method are solved. From the above, the 3D imaging method using multi-line laser scanning in this embodiment is a 3D scanning imaging method with faster detection speed, higher detection accuracy, high precision, low cost, small volume and low power consumption.

[0015] An embodiment of the present invention provides a three-dimensional imaging method using multi-line laser scanning, which may be applied to a three-dimensional imaging device, which may be any device with three-dimensional imaging function, such as any device in the machine vision field or industrial automation field, and the type of the three-dimensional imaging device is not particularly limited.

[0016] 1A is a schematic diagram of a practical configuration of a three-dimensional imaging device, which may include, but is not limited to, a left camera, a right camera, an auxiliary camera, a processor, a laser, a galvanometer motor, and a galvanometer drive. In another configuration of the three-dimensional imaging device, the three-dimensional imaging device may include, but is not limited to, a left camera, a right camera, a processor, a laser, a galvanometer motor, and a galvanometer drive, i.e., does not have an auxiliary camera.

[0017] As shown in FIG. 1B, a block diagram of the configuration of a three-dimensional imaging device is another representation of the three-dimensional imaging device, which may include, but is not limited to, an image acquisition device 100, an opto-mechanical scanning device 200, a multi-line laser emission device 300, an image processing device 400, and a fixing bracket 500.

[0018] For example, the image acquisition device 100 may include a left camera 101 and a right camera 102. Alternatively, the image acquisition device 100 may include a left camera 101, a right camera 102, and an auxiliary camera 103. The left camera 101 and the right camera 102 acquire depth information based on triangulation using a binocular matching multi-line laser. The auxiliary camera 103 may or may not be involved in reconstruction. Here, the left camera 101 and the right camera 102 may be black-and-white cameras, and a filter with the same bandwidth as the laser wavelength may be added to the tip of the camera to pass only light within the laser wavelength range. In other words, by receiving only light of the laser wavelength reflected from the surface of the object to acquire a reflected image of the laser beam, contrast can be improved and interference from ambient light can be reduced. The left camera 101 and the right camera 102 may be mounted symmetrically on both sides of a mechanical galvanometer. The auxiliary camera 103 may be a black-and-white camera or an RGB camera. The auxiliary camera 103 is mounted as close as possible to the optical axis of the mechanical galvanometer, thereby ensuring a short baseline and making the field of view nearly coincident with the laser scanning field of view, so that the auxiliary camera 103 can capture the entire laser beam every time. If the auxiliary camera 103 is an RGB camera, it can turn off the laser and capture a color image of the surface of the object to be measured, thereby realizing the function of outputting an RGBD image.

[0019] For example, the opto-mechanical scanning device 200 may include a mechanical galvanometer. That is, the scanning function is realized using a mechanical galvanometer. The mechanical galvanometer may include three parts: a galvanometer motor, a galvanometer drive, and a mirror. In FIG. 1A, only the galvanometer motor and the galvanometer drive are shown. The mechanical galvanometer may be a mechanical galvanometer with high repeatability. The reflecting mirror has a visible light reflecting film and can reflect the laser beam and change the emission angle of the laser beam.

[0020] For example, the multi-line laser emission device 300 may include a laser. The laser may be a multi-line laser, i.e., a laser that simultaneously emits multiple laser beams. For example, the laser may be a multi-line laser module mainly composed of a laser diode, a collimating lens, and a multi-line DOE (Diffractive Optical Element). The laser diode may be a high-power red laser diode, and the wavelength may be 635 nm, 660 nm, or other wavelengths. The multi-line DOE may simultaneously emit, for example, 10 laser beams, 11 laser beams, or 25 laser beams, but is not limited thereto.

[0021] For example, the image processing device 400 may include a processor such as a CPU or a GPU. The image processing device 400 is connected to the image acquisition device 100, the opto-mechanical scanning device 200, and the multi-line laser emitting device 300. The image processing device 400 activates the multi-line laser emitting device 300, causing the multi-line laser emitting device 300 to emit multiple line lasers (i.e., line structured light), which are reflected by the mechanical galvanometer and then irradiated onto the surface of the workpiece. The image processing device 400 controls the mechanical galvanometer to start scanning, and after scanning one angle, the image processing device 400 may receive angle feedback information from the mechanical galvanometer and trigger the image acquisition device 100 to collect a multi-line laser image of the surface of the workpiece based on the angle feedback information.

[0022] For ease of explanation, in the following embodiments, the image processing device 400 is a processor, the image acquisition device 100 is a left camera, a right camera, and an auxiliary camera, the opto-mechanical scanning device 200 is a mechanical galvanometer, and the multi-line laser emitting device 300 is a laser. As shown in FIG. 1C , the laser can emit multiple line lasers (e.g., seven line lasers in FIG. 1C ), which are reflected by the mechanical galvanometer and then irradiated onto the surface of the object to be measured. The initial angle of the mechanical galvanometer is angle A. At angle A, the left camera collects a line-structured light image A1 of the object to be measured, the right camera collects a line-structured light image A2 of the object to be measured, and the auxiliary camera collects a line-structured light image A3 of the object to be measured. Next, as shown in Figure 1D, the angle of the mechanical galvanometer is angle B, the left camera collects a line-structured light image B1 of the object to be measured, the right camera collects a line-structured light image B2 of the object to be measured, and the auxiliary camera collects a line-structured light image B3 of the object to be measured, repeating in this manner until the final angle of the mechanical galvanometer indicates that a complete scan of the surface of the object to be measured has been completed.

[0023] Based on the line structured light image A1, the line structured light image A2, and the line structured light image A3, the processor can determine three-dimensional points (i.e., a three-dimensional point cloud) at angle A. Based on the line structured light image B1, the line structured light image B2, and the line structured light image B3, the processor can determine three-dimensional points at angle B. By analogy in this way, three-dimensional points at all angles can be obtained. Based on this, the three-dimensional points at all angles can be connected to obtain complete three-dimensional points on the surface of the object to be measured, that is, a complete three-dimensional reconstructed image of the surface of the object to be measured can be obtained.

[0024] If the auxiliary camera is an RGB camera, after completing the scan, the processor may turn off the laser, control the auxiliary camera to collect an RGB image, and then fit and align the 3D reconstructed image and the RGB image to output an RGBD image. The processing process of this RGBD image is not limited in this embodiment.

[0025] For example, the fixing bracket 500 plays the role of fixing and heat dissipation and is made of metal aluminum or other materials. The camera and the mechanical galvanometer adopt an integrated fixing design to ensure that the relative positions of the camera and the mechanical galvanometer do not change.

[0026] The image reconstruction method according to the embodiment of the present invention will be described below with reference to specific examples. Fig. 2 is a flowchart of the image reconstruction method according to the embodiment of the present invention, which may include steps 201 to 206.

[0027] In step 201, when a multi-line laser projects N line structured lights (i.e., laser beams) onto a measurement object (i.e., an object to be measured), a first original image of the measurement object collected by a first camera is obtained, and a second original image of the measurement object collected by a second camera is obtained, where N may be a positive integer greater than 1.

[0028] Exemplarily, the image reconstruction method may be applied to a three-dimensional imaging device. The three-dimensional imaging device may include a first camera, a second camera, and a multi-line laser. The three-dimensional imaging device may further include a processor and a mechanical galvanometer. Here, the first camera may be a left camera and the second camera may be a right camera. Alternatively, the first camera may be a right camera and the second camera may be a left camera. The multi-line laser is the laser of the above embodiment.

[0029] When the angle of the mechanical galvanometer is angle A, the multi-line laser projects N line structured light beams onto the object to be measured, the left camera collects a line structured light image A1 of the object to be measured, and the right camera collects a line structured light image A2 of the object to be measured. Based on this, assuming that the first camera is the left camera and the second camera is the right camera, the processor may obtain a first original image (e.g., line structured light image A1) and a second original image (e.g., line structured light image A2), and perform subsequent processing based on the first original image and the second original image.

[0030] When the angle of the mechanical galvanometer is angle B, the multi-line laser projects N line structured light beams onto the object to be measured, the left camera collects a line structured light image B1 of the object to be measured, and the right camera collects a line structured light image B2 of the object to be measured. Based on this, the processor acquires a first original image (line structured light image B1) and a second original image (line structured light image B2) and performs subsequent processing based on the first original image and the second original image. In this way, at each angle of the mechanical galvanometer, the processor may acquire a first original image of the object to be measured collected by the first camera and a second original image of the object to be measured collected by the second camera.

[0031] For example, when N lines of structured light are projected onto the object to be measured by a multi-line laser, the first original image may include N first light-emitting areas corresponding to the N lines of structured light. The N first light-emitting areas correspond one-to-one to the N lines of structured light. The second original image may include N second light-emitting areas corresponding to the N lines of structured light. The N second light-emitting areas correspond one-to-one to the N lines of structured light.

[0032] In step 202, a first target image corresponding to the first original image and a second target image corresponding to the second original image are determined. The first target image includes N first light-emitting regions corresponding to the N line structured light beams, and the second target image includes N second light-emitting regions corresponding to the N line structured light beams.

[0033] In one possible embodiment, the first original image may be determined as the first target image, and the second original image may be determined as the second target image. Alternatively, binocular correction may be performed on the first original image and the second original image to obtain a first target image corresponding to the first original image and a second target image corresponding to the second original image. Here, the binocular correction is used to make the same position point on the object to be measured have the same pixel height in the first target image and the second target image, and this binocular correction process is not particularly limited.

[0034] In step 203, a first light strip centerline corresponding to each first light-emitting area in the first target image is determined, and a second light strip centerline corresponding to each second light-emitting area in the second target image is determined.

[0035] In step 204, a plurality of keypoint pairs are determined based on all of the first light strip centerlines and all of the second light strip centerlines, each keypoint pair including a first pixel point on the first light strip centerline and a second pixel point on the second light strip centerline, the first pixel point and the second pixel point corresponding to the same position on the object to be measured.

[0036] In one possible embodiment, for each line structured light, a target first light strip centerline and a target second light strip centerline corresponding to the line structured light may be determined from all first light strip centerlines and all second light strip centerlines, and for each first pixel point on the target first light strip centerline, the first pixel point may be projected onto a second target image to obtain a projected pixel point corresponding to the first pixel point, and a second pixel point may be selected from the target second light strip centerline corresponding to the projected pixel point, and a keypoint pair may be generated based on the first pixel point and the second pixel point, i.e., the keypoint pair includes the first pixel point and the second pixel point.

[0037] Exemplarily, selecting a second pixel point corresponding to the projected pixel point from the target second light strip center line includes, but is not limited to, determining a pixel point from the target second light strip center line that has the same pixel height as the projected pixel point, and if the determined pixel point is one, selecting the pixel point as the second pixel point; and if the determined pixel points are at least two, determining reprojection errors between the at least two pixel points and the projected pixel point, and selecting the pixel point corresponding to the smallest reprojection error as the second pixel point.

[0038] Exemplarily, projecting a first pixel point onto a second target image and obtaining a projected pixel point corresponding to the first pixel point includes, but is not limited to, obtaining a first calibration equation and a second calibration equation corresponding to the line structured light, transforming the first pixel point into a target 3D reconstruction point based on the first calibration equation, and transforming the target 3D reconstruction point into the projected pixel point based on the second calibration equation, where the first calibration equation represents a functional relationship between the pixel point in the first target image and the 3D reconstruction point, and the second calibration equation represents a functional relationship between the pixel point in the second target image and the 3D reconstruction point.

[0039] In step 205, a 3D point corresponding to the keypoint pair is determined based on the keypoint pair and the camera calibration parameters.

[0040] In one possible embodiment, the camera calibration parameters may include intrinsic camera parameters of the first camera, intrinsic camera parameters of the second camera, and extrinsic camera parameters between the first camera and the second camera. Distortion correction may be performed on a first pixel point using the intrinsic camera parameters of the first camera, and the corrected pixel point may be transformed into first homogeneous coordinates. Distortion correction may be performed on a second pixel point using the intrinsic camera parameters of the second camera, and the corrected pixel point may be transformed into second homogeneous coordinates. Then, a 3D point corresponding to the keypoint pair is determined by a triangulation method based on the first homogeneous coordinates, the second homogeneous coordinates, the intrinsic camera parameters of the first camera, the intrinsic camera parameters of the second camera, and the extrinsic camera parameters. The triangulation method is not limited.

[0041] In step 206, a 3D reconstruction of the object is generated based on the 3D points corresponding to the plurality of keypoint pairs.

[0042] For example, when the angle of the mechanical galvanometer is angle A, three-dimensional points corresponding to a plurality of key point pairs may be determined based on the first and second original images corresponding to angle A. When the angle of the mechanical galvanometer is angle B, three-dimensional points corresponding to a plurality of key point pairs may be determined based on the first and second original images corresponding to angle B. In this way, a three-dimensional reconstructed image can be generated based on three-dimensional points corresponding to all angles, that is, a complete three-dimensional reconstructed image of the surface of the object to be measured can be obtained.

[0043] In one possible embodiment, the three-dimensional imaging device may further include a third camera, i.e., an auxiliary camera. Based on this, when the multi-line laser projects N lines of structured light onto the object to be measured, a third original image of the object to be measured (e.g., line structured light image A3, line structured light image B3, etc.) collected by the third camera may be obtained, and a third target image corresponding to the third original image may be determined. The third target image includes N third light-emitting areas corresponding to the N lines of structured light. A third light strip centerline corresponding to each third light-emitting area in the third target image is determined. For each line structured light, a target third light strip centerline corresponding to the line structured light is determined from all the third light strip centerlines. For each first pixel point on the target first light strip centerline, projecting the first pixel point onto the second target image to obtain a projected pixel point corresponding to the first pixel point may further include: determining a third pixel point from the target third light strip centerline having the same pixel height as the first pixel point; determining a target 3D reconstruction point based on the first pixel point, the third pixel point, and camera calibration parameters; and transforming the target 3D reconstruction point into the projected pixel point based on a third calibration equation, where the third calibration equation represents a functional relationship between the pixel point in the second target image and the 3D reconstruction point.

[0044] In an embodiment of the present invention, the multi-line laser projects N line structured light beams onto the object to be measured each time, where N is a positive integer greater than 1, such as 7, 11, or 15. Therefore, the line structured light image collected by the camera each time includes N light strip center lines. The line structured light image is equivalent to line structured light images of N positions on the object to be measured. This reduces the number of times the line structured light images need to be collected and the time required for 3D reconstruction. When the multi-line laser scans the surface of the object to be measured, the entire contour data of the object can be quickly acquired, and 3D image information of the object can be output, improving detection accuracy and speed. In addition, the first camera and the second camera are used to simultaneously collect line-structured light images, and based on the line-structured light images collected by the two cameras, three-dimensional information of the object to be measured can be obtained by triangulation, that is, depth information of the object to be measured can be obtained. Thus, depth information in multi-line laser light can be obtained by a single collected image, the efficiency of a single scan can be improved by N times, and a full-size scan of the entire contour of the object to be measured can be quickly achieved.

[0045] The above technical solutions of the embodiments of the present invention will be described below with reference to specific application scenarios.

[0046] Application Scenario 1: The three-dimensional imaging device may include a first camera, a second camera, a processor, a multi-line laser, and a mechanical galvanometer. The first camera is a left camera and the second camera is a right camera, or the first camera is a right camera and the second camera is a left camera. In Application Scenario 1, camera calibration parameters, a first calibration equation, and a second calibration equation corresponding to the three-dimensional imaging device may be acquired in advance, and the camera calibration parameters, the first calibration equation, and the second calibration equation may be stored in the three-dimensional imaging device.

[0047] For example, the camera calibration parameters may include intrinsic camera parameters of a first camera, intrinsic camera parameters of a second camera, and extrinsic camera parameters between the first camera and the second camera. The intrinsic camera parameters of the first camera are parameters related to the characteristics of the first camera itself, such as focal length, pixel size, and distortion coefficient. The intrinsic camera parameters of the second camera are parameters related to the characteristics of the second camera itself, such as focal length, pixel size, and distortion coefficient. The extrinsic camera parameters between the first camera and the second camera are parameters in a world coordinate system, such as the position and rotation direction of the first camera, the position and rotation direction of the second camera, and the positional relationship between the first camera and the second camera, such as a rotation matrix and a translation matrix.

[0048] The intrinsic camera parameters of the first camera are parameters inherent to the first camera. The intrinsic camera parameters of the first camera are already given when the first camera is shipped. The intrinsic camera parameters of the second camera are parameters inherent to the second camera. The intrinsic camera parameters of the second camera are already given when the second camera is shipped.

[0049] For camera extrinsic parameters between a first camera and a second camera, such as a rotation matrix and a translation matrix, a plurality of calibration points are arranged in a target scene, a first calibration image of the target scene is collected by the first camera, the first calibration image includes the plurality of calibration points, and a second calibration image of the target scene is collected by the second camera, the second calibration image includes the plurality of calibration points. The camera extrinsic parameters between the first camera and the second camera can be determined based on pixel coordinates of the plurality of calibration points in the first calibration image and pixel coordinates of the plurality of calibration points in the second calibration image, and the process of determining the camera extrinsic parameters is not limited.

[0050] For example, the first calibration equation represents a functional relationship between pixel points in an image (referred to as image s1) collected by a first camera and three-dimensional reconstruction points, and the second calibration equation represents a functional relationship between pixel points in an image (referred to as image s2) collected by a second camera and three-dimensional reconstruction points. Assuming that a multi-line laser projects N lines of structured light onto the object to be measured and that the mechanical galvanometer has a total of M angles, a total of N×M first calibration equations and N×M second calibration equations need to be obtained, and both the first calibration equation and the second calibration equation may be light plane equations. The method for obtaining the first calibration equation and the second calibration equation may include steps S11 to S15.

[0051] In step S11, when the multi-line laser projects N lines of structured light onto a white background plate for each angle of the mechanical galvanometer, an image s1 is acquired by a first camera and an image s2 is acquired by a second camera. Here, the image s1 includes N first light-emitting areas corresponding to the N lines of structured light, and the N first light-emitting areas correspond one-to-one to the N lines of structured light. The image s2 includes N second light-emitting areas corresponding to the N lines of structured light, and the N second light-emitting areas correspond one-to-one to the N lines of structured light.

[0052] In step S12, a first light strip centerline corresponding to each first light-emitting area in image s1 is determined, and a second light strip centerline corresponding to each second light-emitting area in image s2 is determined, that is, N first light strip centerlines corresponding to N line structured lights are obtained, and N second light strip centerlines corresponding to N line structured lights are obtained.

[0053] In step S13, a plurality of keypoint pairs are determined based on all the first light strip centerlines and all the second light strip centerlines, each keypoint pair including a first center point on the first light strip centerline and a second center point on the second light strip centerline, where the first center point and the second center point are pixel points corresponding to the same position on the white background board.

[0054] For example, assume that the N line structured lights are line structured light 1 and line structured light 2, image s1 includes a first light-emitting region 1 corresponding to line structured light 1 and a first light-emitting region 2 corresponding to line structured light 2, and image s2 includes a second light-emitting region 1 corresponding to line structured light 1 and a second light-emitting region 2 corresponding to line structured light 2. The first light-emitting region 1 corresponds to the first light strip centerline 1, the first light-emitting region 2 corresponds to the first light strip centerline 2, the second light-emitting region 1 corresponds to the second light strip centerline 1, and the second light-emitting region 2 corresponds to the second light strip centerline 2.

[0055] For example, when the object to be measured is a white background plate and two line structured lights are projected onto the white background plate, the light-emitting areas are relatively clear and noise-free. Therefore, when determining the first light strip centerline 1 based on the first light-emitting area 1, each row of the first light strip centerline 1 has only one center point. Similarly, each row of the second light strip centerline 1 has only one center point. Based on this, the center point of the first row of the first light strip centerline 1 and the center point of the first row of the second light strip centerline 1 are configured as key point pair 11, and the center point of the second row of the first light strip centerline 1 and the center point of the second row of the second light strip centerline 1 are configured as key point pair 12. Similarly, the center point of the first row of the first light strip centerline 2 and the center point of the first row of the second light strip centerline 2 are configured as key point pair 21, and the center point of the second row of the first light strip centerline 2 and the center point of the second row of the second light strip centerline 2 are configured as key point pair 22, and by this analogy.

[0056] In step S14, for each keypoint pair, a 3D point corresponding to the keypoint pair is determined based on the keypoint pair and the camera calibration parameters. For example, a triangulation method may be used to determine the 3D point corresponding to the keypoint pair. The triangulation method can be referred to in subsequent embodiments, and its description will be omitted here.

[0057] In step S15, a first calibration equation and a second calibration equation corresponding to the angle of the mechanical galvanometer and the line structured light are determined based on the keypoint pairs and the three-dimensional points corresponding to the keypoint pairs.

[0058] For example, for an angle A of a mechanical galvanometer, a first calibration equation and a second calibration equation corresponding to the angle A and the line structured light 1 are determined based on a plurality of key point pairs (e.g., key point pair 11, key point pair 12, etc.) between the first light strip center line 1 and the second light strip center line 1, and three-dimensional points corresponding to each key point pair.

[0059] For example, a first calibration equation may be determined based on a large number of center points of the first light strip centerline 1 and the three-dimensional points corresponding to each center point. The first calibration equation is used to represent the functional relationship between pixel points in the image s1 (i.e., the center points of the first light strip centerline 1) and the three-dimensional reconstruction points (i.e., the three-dimensional points corresponding to the center points). For example, the first calibration equation may be obtained by fitting using a planar model or a quadratic model.

[0060] A second calibration equation may be determined based on a large number of center points of the second light strip centerline 1 and the three-dimensional points corresponding to each center point. The second calibration equation is used to express the functional relationship between pixel points in image s2 (i.e., center points of the second light strip centerline 1) and three-dimensional reconstruction points (i.e., three-dimensional points corresponding to the center points).

[0061] Similarly, a first calibration equation and a second calibration equation corresponding to angle A and line structured light 2 can be obtained, and a first calibration equation and a second calibration equation corresponding to angle B and line structured light 1 can be obtained, and so on.

[0062] As a result of the above, for each angle of the mechanical galvanometer, the first calibration equation and the second calibration equation corresponding to each line structured light, i.e., N×M first calibration equations and N×M second calibration equations, can be obtained.

[0063] For example, for N line structured lights projected by a multi-line laser, each line structured light may be numbered according to an actual order, for example, N line structured lights (i.e., laser beams) may be denoted as 1, 2, 3, ..., N according to the order from left to right (or from right to left), which makes it easy to match and index each line structured light.

[0064] In the above application scenario 1, as shown in FIG. 3, the image reconstruction method of this embodiment may include steps 301 to 309.

[0065] In step 301, when a multi-line laser projects N line structured light beams onto an object to be measured, a first original image of the object to be measured is acquired by a first camera, and a second original image of the object to be measured is acquired by a second camera. The acquisition time of the first original image and the acquisition time of the second original image may be the same.

[0066] Illustratively, the first original image includes N first light-emitting regions corresponding to the N line structured lights, for example, first light-emitting region 1 corresponding to line structured light 1, first light-emitting region 2 corresponding to line structured light 2, ... etc. The second original image includes N second light-emitting regions corresponding to the N line structured lights, for example, second light-emitting region 1 corresponding to line structured light 1, second light-emitting region 2 corresponding to line structured light 2, ... etc.

[0067] In step 302, binocular correction is performed on the first original image and the second original image to obtain a first target image corresponding to the first original image and a second target image corresponding to the second original image.

[0068] For example, binocular correction is used to ensure that the same location on the object under measurement has the same pixel height in the first target image and the second target image. In other words, binocular correction corrects the first and second original images to the same pixel height for the same location on the object under measurement, allowing matching to be performed directly within a single row, making matching more convenient. For example, matching corresponding points in two-dimensional space can be very time-consuming. To reduce the search range for matching, epipolar constraints can be used to reduce the matching process from a two-dimensional search to a one-dimensional search. The function of binocular correction is to match the first and second original images row by row to obtain the first and second target images. This ensures that the epipolar lines of the first and second target images are exactly horizontal, ensuring that any point on the first target image and its corresponding point on the second target image always have the same row number, allowing for one-dimensional search within that row.

[0069] Illustratively, the first target image includes N first light-emitting regions corresponding to the N line structured lights, for example, first light-emitting region 1 corresponding to line structured light 1, first light-emitting region 2 corresponding to line structured light 2, ... etc. The second target image includes N second light-emitting regions corresponding to the N line structured lights, for example, second light-emitting region 1 corresponding to line structured light 1, second light-emitting region 2 corresponding to line structured light 2, ... etc.

[0070] In step 303, a first light strip centerline corresponding to each first light-emitting area in the first target image is determined, and a second light strip centerline corresponding to each second light-emitting area in the second target image is determined.

[0071] For example, each row of first light-emitting areas may include multiple pixel points, and a center point of the row may be selected from the multiple pixel points. The center points of all rows of first light-emitting areas constitute a first light strip centerline, thereby obtaining a first light strip centerline 1 corresponding to first light-emitting area 1, a first light strip centerline 2 corresponding to first light-emitting area 2, etc. Similarly, a second light strip centerline 1 corresponding to second light-emitting area 1, a second light strip centerline 2 corresponding to second light-emitting area 2, etc.

[0072] For example, a light strip centerline extraction algorithm may be used to determine the light strip centerline corresponding to the light emitting area. For example, Gaussian fitting, COG (Center of Gravity), STEGER, or other methods may be used to extract the center point of each row of the light emitting area to obtain the light strip centerline, and this embodiment is not limited thereto.

[0073] For example, if the height of the first target image and the second target image is H, each of the first light strip center lines includes the center point of the H row, and each of the second light strip center lines includes the center point of the H row.

[0074] In step 304, for each line structured light, a target first light strip centerline and a target second light strip centerline corresponding to the line structured light are determined from all the first light strip centerlines and all the second light strip centerlines.

[0075] For example, the first light strip centerline 1 and the second light strip centerline 1 corresponding to the line structured light 1 are determined, and the first light strip centerline 2 and the second light strip centerline 2 corresponding to the line structured light 2 are determined, and this is used for inference.

[0076] In step 305, for each line structured light, based on a first calibration equation corresponding to the line structured light and a target first light strip center line corresponding to the line structured light, for each first pixel point on the target first light strip center line, the first pixel point is converted into a target three-dimensional reconstruction point based on the first calibration equation.

[0077] For example, the angle of the mechanical galvanometer may be determined, i.e., the angle at which the first original image and the second original image were collected may be determined. For each line structured light, a first calibration equation corresponding to the angle and the line structured light may be selected from N×M first calibration equations. The first calibration equation represents a functional relationship between pixel points in the first target image and three-dimensional reconstruction points, so that each first pixel point on the target first light strip centerline can be converted to a target three-dimensional reconstruction point based on the first calibration equation.

[0078] For example, for a first light strip centerline 1 corresponding to the line structured light 1, each first pixel point on the first light strip centerline 1 is converted into a target three-dimensional reconstruction point based on a first calibration equation corresponding to the line structured light 1. For a first light strip centerline 2 corresponding to the line structured light 2, each first pixel point on the first light strip centerline 2 is converted into a target three-dimensional reconstruction point based on a first calibration equation corresponding to the line structured light 2, by analogy.

[0079] In step 306, for each target 3D reconstruction point corresponding to each first pixel point, the target 3D reconstruction point may be converted into a projected pixel point in a second target image based on a second calibration equation corresponding to the line structured light, where the projected pixel point is a projected pixel point corresponding to the first pixel point.

[0080] For example, for each line structured light, a second calibration equation corresponding to the line structured light is selected from the N×M second calibration equations, and the second calibration equation represents a functional relationship between pixel points in the second target image and three-dimensional reconstruction points, so that after the first pixel points on the target first light strip center line are transformed into target three-dimensional reconstruction points, the target three-dimensional reconstruction points can be transformed into projected pixel points based on the second calibration equation.

[0081] For example, when converting the target 3D reconstruction points into projected pixel points based on the second calibration equation corresponding to line structured light 1, projected pixel points corresponding to each first pixel point on the first light strip center line 1 are obtained. Also, when converting the target 3D reconstruction points into projected pixel points based on the second calibration equation corresponding to line structured light 2, projected pixel points corresponding to each first pixel point on the first light strip center line 2 are obtained, and so on.

[0082] In summary, for each first pixel point on the target first light strip centerline, the first pixel point can be projected onto the second target image to obtain a projected pixel point corresponding to the first pixel point.

[0083] In step 307, for each first pixel point, after obtaining the projected pixel point corresponding to the first pixel point, select the second pixel point corresponding to the projected pixel point from the target second light strip centerline.

[0084] For example, for each line structured light, taking the line structured light 1 as an example, a target second light strip center line corresponding to the line structured light 1, i.e., the second light strip center line 1, may be determined. For each first pixel point on the first light strip center line 1 corresponding to the line structured light 1, after obtaining a projected pixel point corresponding to the first pixel point, a second pixel point corresponding to the projected pixel point may be selected from the second light strip center line 1.

[0085] Obviously, the first pixel point and the second pixel point may constitute a keypoint pair, that is, the keypoint pair includes a first pixel point on the first light strip center line 1 and a second pixel point on the second light strip center line 1, and the first pixel point and the second pixel point are pixel points corresponding to the same position on the object to be measured. The first pixel point is a pixel point in the first target image, and the second pixel point is a pixel point in the second target image.

[0086] In one possible embodiment, selecting a second pixel point from the target second light strip center line that corresponds to the projected pixel point may include determining a pixel point having the same pixel height from the target second light strip center line as the projected pixel point, and if there is one determined pixel point, selecting the pixel point as the second pixel point; and if there are at least two determined pixel points, determining reprojection errors between the at least two pixel points and the projected pixel point, and selecting the pixel point corresponding to the smallest reprojection error as the second pixel point.

[0087] For example, a row of the center line of the second light strip may include one pixel point. In this case, if there is one pixel point with the same pixel height as the projected pixel point, the pixel point is selected as the second pixel point. Alternatively, a row of the center line of the second light strip may include at least two pixel points. For example, if there is noise in the luminous area, there will be at least two pixel points in a row. In this case, if there are at least two pixel points with the same pixel height as the projected pixel point, the reprojection error between the projected pixel point and each pixel point is determined. The method of this determination is not particularly limited. After obtaining the reprojection errors between the projected pixel point and each pixel point, the pixel point corresponding to the smallest reprojection error may be selected as the second pixel point.

[0088] In step 308, a 3D point corresponding to the keypoint pair is determined based on the keypoint pair and the camera calibration parameters.

[0089] For example, for each keypoint pair, the keypoint pair includes a first pixel point in a first target image and a second pixel point in a second target image, and the first pixel point and the second pixel point are pixel points corresponding to the same position on the object to be measured. Based on this, a 3D point corresponding to the keypoint pair can be determined by triangulation. The following describes this process by combining specific steps.

[0090] In step 3081, distortion correction is performed on a first pixel point using the internal camera parameters of the first camera, and the pixel point after distortion correction is transformed into first homogeneous coordinates. Distortion correction is performed on a second pixel point using the internal camera parameters of the second camera, and the pixel point after distortion correction is transformed into second homogeneous coordinates.

[0091] For example, due to variations in lens manufacturing accuracy and assembly process, distortions such as radial distortion and tangential distortion exist in the image collected by the first camera. To solve the distortion problem, the intrinsic camera parameters of the first camera include distortion parameters such as radial distortion parameters k1, k2, and k3 and tangential distortion parameters p1 and p2. Based on this, in this embodiment, distortion correction may be performed on the first pixel point using the intrinsic camera parameters of the first camera to obtain pixel coordinates after distortion removal processing. After obtaining the pixel coordinates after distortion removal processing, the pixel coordinates after distortion removal processing may be converted into first homogeneous coordinates. Similarly, the intrinsic camera parameters of the second camera may be used to correct the second pixel point. point to obtain distortion-corrected pixel coordinates, and then transform the distortion-corrected pixel coordinates into second homogeneous coordinates. In summary, the homogeneous coordinates of a keypoint pair may include a first homogeneous coordinate of a first keypoint and a second homogeneous coordinate of a second keypoint.

[0092] In step 3082, a three-dimensional point corresponding to the keypoint pair is determined by triangulation based on the first homogeneous coordinates, the second homogeneous coordinates, the internal camera parameters of the first camera, the internal camera parameters of the second camera, and the external camera parameters (e.g., positional relationship, etc.) between the first camera and the second camera.

[0093] For example, Figure 4 shows a schematic diagram of the triangulation method. L is the position of the first camera, and O R is the position of the second camera, and based on the camera extrinsic parameters between the first and second cameras, O L and O R For a three-dimensional point P in three-dimensional space, the imaging position on the image plane of the first camera is p l and the imaging position on the image plane of the second camera is p r p l Let p be the first pixel point. ris the second pixel point, and the first pixel point and the second pixel point form a keypoint pair. The 3D point P is the 3D point corresponding to the keypoint pair. L , O R , p l and p r Transform into the same coordinate system, and O in the same coordinate system L , O R , p l and p r About O L and p l There is a line a1 between R and p r There is a line a2 between the lines a1 and a2. If an intersection point exists between the lines a1 and a2, the intersection point between the lines a1 and a2 is a 3D point P. If an intersection point does not exist between the lines a1 and a2, the 3D point P is the point closest to the lines a1 and a2. Based on the above application scenario, the 3D space coordinate of the 3D point P can be obtained by a triangulation method, thereby obtaining a 3D point corresponding to the keypoint pair. Of course, the above embodiment is merely an example of a triangulation method, and the embodiment of the triangulation method is not limited.

[0094] In summary, for each keypoint pair, a 3D point corresponding to the keypoint pair can be obtained, and since the first target image includes N first light strip center lines, and each first light strip center line includes H first pixel points, N×H keypoint pairs can be obtained, and the N×H keypoint pairs correspond to N×H 3D points.

[0095] In step 309, a 3D reconstructed image is generated based on the 3D points corresponding to the plurality of keypoint pairs.

[0096] For example, for each angle of the mechanical galvanometer, N×H three-dimensional points at that angle may be determined using steps 301 to 308. During the scanning process of the mechanical galvanometer, a set of original images may be acquired at each angle, and the above operation may be performed. Assuming that the mechanical galvanometer has a total of M angles, M×N×H three-dimensional points at the M angles are obtained. Based on this, a three-dimensional reconstructed image, i.e., point cloud data, may be generated based on the M×N×H three-dimensional points, and the three-dimensional reconstructed image may be output. Alternatively, the three-dimensional reconstructed image may be projected onto a camera to acquire a depth image, and the depth image may be output.

[0097] Application Scenario 2: The three-dimensional imaging device may include a first camera, a second camera, a third camera, a processor, a multi-line laser, and a mechanical galvanometer. The first camera is a left camera, the second camera is a right camera, and the third camera is an auxiliary camera, or the first camera is a right camera, the second camera is a left camera, and the third camera is an auxiliary camera. Camera calibration parameters and a third calibration equation corresponding to the three-dimensional imaging device may be obtained in advance, and the camera calibration parameters and the third calibration equation may be stored in the three-dimensional imaging device.

[0098] Illustratively, the camera calibration parameters include camera internal parameters of the first camera, camera internal parameters of the second camera, camera internal parameters of the third camera, camera external parameters between the first camera and the second camera (e.g., positional relationships such as rotation matrices and translation matrices), camera external parameters between the first camera and the third camera (e.g., positional relationships), and camera external parameters between the second camera and the third camera (e.g., positional relationships).

[0099] For the method of obtaining camera calibration parameters, refer to Application Scenario 1, and the explanation will be omitted here.

[0100] For example, the third calibration equation represents the functional relationship between pixel points in the image (referred to as image s2) collected by the second camera and the 3D reconstruction points. Assuming that the multi-line laser projects N line structured light beams onto the object to be measured and the mechanical galvanometer has a total of M angles, a total of N × M third calibration equations need to be obtained, and each of the third calibration equations may be a light plane equation. Regarding the method for obtaining the third calibration equations, reference may be made to the method for obtaining the second calibration equation in application scenario 1, and the description thereof will be omitted here.

[0101] For example, for N line structured lights projected by a multi-line laser, each line structured light may be numbered according to an actual order, for example, N line structured lights (i.e., laser beams) may be denoted as 1, 2, 3, ..., N according to the order from left to right (or from right to left), which makes it easy to match and index each line structured light.

[0102] In the above application scenario 2, as shown in FIG. 5, the image reconstruction method of this embodiment may include steps 501 to 509.

[0103] In step 501, when a multi-line laser projects N lines of structured light onto an object to be measured, a first original image of the object to be measured is acquired by a first camera, a second original image of the object to be measured is acquired by a second camera, and a third original image of the object to be measured is acquired by a third camera. The acquisition time of the first original image, the acquisition time of the second original image, and the acquisition time of the third original image may be the same.

[0104] For example, the first original image may include N first light-emitting regions corresponding to N line structured lights, the second original image may include N second light-emitting regions corresponding to N line structured lights, and the third original image may include N third light-emitting regions corresponding to N line structured lights.

[0105] In step 502, trinocular correction is performed on the first original image, the second original image, and the third original image to obtain a first target image corresponding to the first original image, a second target image corresponding to the second original image, and a third target image corresponding to the third original image. Here, trinocular correction is used to ensure that the same position point on the object to be measured has the same pixel height in the first target image, the second target image, and the third target image. That is, for the same position point on the object to be measured, the first original image, the second original image, and the third original image can be corrected to have the same pixel height by trinocular correction.

[0106] For example, the first target image may include N first light-emitting regions corresponding to N line structured lights, the second target image may include N second light-emitting regions corresponding to N line structured lights, and the third target image may include N third light-emitting regions corresponding to N line structured lights.

[0107] In step 503, a first light strip centerline corresponding to each first light-emitting area in the first target image is determined, a second light strip centerline corresponding to each second light-emitting area in the second target image is determined, and a third light strip centerline corresponding to each third light-emitting area in the third target image is determined.

[0108] For example, a light strip centerline extraction algorithm may be used to determine the light strip centerline corresponding to the light emitting area. For example, Gaussian fitting, COG, or STEGER may be used to extract the center point of each row of the light emitting area to obtain the light strip centerline, and this embodiment is not limited thereto.

[0109] In step 504, for each line structured light, a target first light strip centerline corresponding to the line structured light is determined from all first light strip centerlines, a target second light strip centerline corresponding to the line structured light is determined from all second light strip centerlines, and a target third light strip centerline corresponding to the line structured light is determined from all third light strip centerlines.

[0110] In step 505, based on the target first light strip center line and the target third light strip center line corresponding to the line structured light, for each first pixel point on the target first light strip center line, a third pixel point having the same pixel height from the target third light strip center line as the first pixel point is determined, and a target three-dimensional reconstruction point corresponding to the first pixel point is determined based on the first pixel point, the third pixel point and camera calibration parameters.

[0111] For example, for each first pixel point on the target first light strip center line, a pixel point having the same pixel height as the first pixel point from the target third light strip center line is determined. If there is one determined pixel point, the pixel point is selected as the third pixel point. If there are at least two determined pixel points, the reprojection errors between the at least two pixel points and the first pixel point are determined, and the pixel point corresponding to the smallest reprojection error is selected as the third pixel point. The manner of determining the reprojection error is not limited.

[0112] For example, the first pixel point and the third pixel point may constitute a keypoint pair. That is, the keypoint pair includes a first pixel point in a first target image and a third pixel point in a third target image, and the first pixel point and the third pixel point correspond to the same position on the target object. Based on this, a 3D point corresponding to the keypoint pair can be determined by triangulation, and the 3D point is the target 3D reconstruction point corresponding to the first pixel point. For example, the first pixel point is corrected for distortion using the intrinsic camera parameters of the first camera, and the corrected pixel point is transformed into first homogeneous coordinates. The third pixel point is corrected for distortion using the intrinsic camera parameters of the third camera, and the corrected pixel point is transformed into third homogeneous coordinates. A three-dimensional point corresponding to the keypoint pair is determined by triangulation based on the first homogeneous coordinates, the third homogeneous coordinates, the internal camera parameters of the first camera, the internal camera parameters of the third camera, and the external camera parameters (e.g., positional relationship, etc.) between the first camera and the third camera.

[0113] In summary, for each first pixel point on the target first light strip centerline, a target 3D reconstruction point corresponding to the first pixel point is determined, and a correspondence relationship between the first pixel point and the target 3D reconstruction point is obtained.

[0114] In step 506, for each target 3D reconstruction point corresponding to each first pixel point, the target 3D reconstruction point may be converted into a projected pixel point in a second target image based on a third calibration equation corresponding to the line structured light, where the projected pixel point is a projected pixel point corresponding to the first pixel point.

[0115] For example, for each line structured light, a third calibration equation corresponding to the line structured light is selected from the N×M third calibration equations, and the third calibration equation represents a functional relationship between the pixel points in the second target image and the three-dimensional reconstruction points, so that after the first pixel points on the target first light strip center line are transformed into the target three-dimensional reconstruction points, the target three-dimensional reconstruction points can be transformed into projected pixel points based on the third calibration equation.

[0116] In summary, for each first pixel point on the target first light strip centerline, the first pixel point can be projected onto the second target image to obtain a projected pixel point corresponding to the first pixel point.

[0117] In step 507, for each first pixel point, after obtaining the projected pixel point corresponding to the first pixel point, select the second pixel point corresponding to the projected pixel point from the target second light strip centerline.

[0118] For example, a pixel point having the same pixel height as the projected pixel point from the center line of the target second light strip may be determined, and if there is one determined pixel point, the pixel point may be selected as the second pixel point; if there are at least two determined pixel points, the reprojection error between the at least two pixel points and the projected pixel point may be determined, and the pixel point corresponding to the smallest reprojection error may be selected as the second pixel point.

[0119] Obviously, the first pixel point and the second pixel point may constitute a keypoint pair, where the first pixel point and the second pixel point correspond to the same position on the object to be measured, and the first pixel point is a pixel point in the first target image, and the second pixel point is a pixel point in the second target image.

[0120] In step 508, a 3D point corresponding to the keypoint pair is determined based on the keypoint pair and the camera calibration parameters. For example, a first pixel point is corrected for distortion using the intrinsic camera parameters of the first camera, and the corrected pixel point is transformed into a first homogeneous coordinate system. A second pixel point is corrected for distortion using the intrinsic camera parameters of the second camera, and the corrected pixel point is transformed into a second homogeneous coordinate system. A 3D point corresponding to the keypoint pair is determined by triangulation based on the first homogeneous coordinate system, the second homogeneous coordinate system, the intrinsic camera parameters of the first camera, the intrinsic camera parameters of the second camera, and the extrinsic camera parameters between the first and second cameras.

[0121] In summary, for each keypoint pair, we can obtain the 3D point that corresponds to that keypoint pair.

[0122] In step 509, a 3D reconstructed image is generated based on the 3D points corresponding to the plurality of keypoint pairs.

[0123] In an embodiment of the present invention, a multi-line laser projects N lines of structured light onto the object to be measured each time, and the line-structured light image collected by the camera each time includes N center lines of light strips. The line-structured light image is equivalent to line-structured light images of N positions on the object to be measured. This reduces the number of times line-structured light images need to be collected and the time required for 3D reconstruction. When scanning the surface of the object with a multi-line laser, the entire contour data of the object can be quickly obtained, and 3D image information of the object can be output, improving detection accuracy and speed. Furthermore, a first camera and a second camera can simultaneously collect line-structured light images. Based on the line-structured light images collected by the two cameras, 3D information of the object can be obtained by triangulation, i.e., depth information of the object can be obtained. This allows depth information of the multi-line laser light to be obtained with a single collected image, improving the efficiency of a single scan by N times, and quickly achieving a full-size scan of the entire contour of the object to be measured. By using multi-line laser triangulation to simultaneously obtain depth information from multiple laser beams and using a mechanical galvanometer to achieve scanning with small angle intervals between the laser beams, it is possible to complete a highly accurate scan of the entire surface contour of the object to be measured. The laser has high contrast, good collimation, a relatively large depth of field, better adaptability to materials in 3D detection applications, and higher detection accuracy, making it suitable for 3D measurement applications in machine vision and 3D grasping and positioning applications in industrial automation.

[0124] Based on the same concept as the above method, an embodiment of the present invention provides an image reconstruction device applied to a three-dimensional imaging device, the three-dimensional imaging device including a first camera, a second camera, and a multi-line laser. As shown in Figure 6, a schematic diagram of the device is shown, and the device may include an acquisition module 61, a determination module 62, and a generation module 63.

[0125] The acquisition module 61 is configured to acquire a first original image of the object to be measured collected by the first camera and acquire a second original image of the object to be measured collected by the second camera when the multi-line laser projects N lines of structured light onto the object to be measured, where N is a positive integer greater than 1.

[0126] The determination module 62 is configured to determine a first target image corresponding to a first original image and a second target image corresponding to a second original image, determine first light strip centerlines corresponding to each first light-emitting area in the first target image, determine second light strip centerlines corresponding to each second light-emitting area in the second target image, determine a plurality of keypoint pairs based on all the first light strip centerlines and all the second light strip centerlines, and determine three-dimensional points corresponding to the keypoint pairs based on the keypoint pairs and camera calibration parameters, where the first target image includes N first light-emitting areas corresponding to the N line structured lights, the second target image includes N second light-emitting areas corresponding to the N line structured lights, and for each keypoint pair, the keypoint pair includes a first pixel point on the first light strip centerline and a second pixel point on the second light strip centerline, and the first pixel point and the second pixel point are pixel points corresponding to the same position on the object to be measured.

[0127] The generation module 63 is configured to generate a 3D reconstructed image of the object based on the 3D points corresponding to the plurality of keypoint pairs.

[0128] Illustratively, when determining a first target image corresponding to a first original image and a second target image corresponding to a second original image, the determining module 62 specifically: configured to determine the first original image as the first target image and the second original image as the second target image, or configured to perform binocular correction on a first original image and a second original image, and obtain a first target image corresponding to the first original image and a second target image corresponding to the second original image; The binocular correction is used to ensure that the same position point on the object to be measured has the same pixel height in the first target image and the second target image.

[0129] Illustratively, when determining a plurality of key point pairs based on all the first light strip centerlines and all the second light strip centerlines, the determining module 62 specifically: For each line structured light, determine a target first light strip centerline and a target second light strip centerline corresponding to the line structured light from all of the first light strip centerlines and all of the second light strip centerlines; For each first pixel point on the target first light strip centerline, project the first pixel point onto the second target image to obtain a projected pixel point corresponding to the first pixel point; and select a second pixel point from the target second light strip centerline corresponding to the projected pixel point; A keypoint pair is configured to be generated based on the first pixel point and the second pixel point.

[0130] Illustratively, when selecting the second pixel point corresponding to the projected pixel point from the target second light strip center line, the determining module 62 specifically: determining a pixel point from the target second light strip centerline that has the same pixel height as the projected pixel point; If the determined pixel point is one, select the pixel point as the second pixel point; If the determined pixel points are at least two, the method is configured to determine a reprojection error between the at least two pixel points and the projected pixel point, and select the pixel point corresponding to the smallest reprojection error as the second pixel point.

[0131] Exemplarily, when the determination module 62 projects the first pixel point onto the second target image to obtain a projected pixel point corresponding to the first pixel point, the determination module 62 specifically: obtaining a first calibration equation and a second calibration equation corresponding to the line structured light; transforming the first pixel points into target 3D reconstruction points based on the first calibration equation; configured to transform the target 3D reconstruction points into the projected pixel points based on the second calibration equation; The first calibration equation represents a functional relationship between pixel points in a first target image and three-dimensional reconstruction points, and the second calibration equation represents a functional relationship between pixel points in a second target image and three-dimensional reconstruction points.

[0132] Illustratively, the three-dimensional imaging device further includes a third camera; The acquisition module 61 is further configured to acquire a third original image of the object to be measured, which is collected by a third camera, when a multi-line laser projects N lines of structured light onto the object to be measured; The determination module 62 is further configured to determine a third target image corresponding to the third original image; determine third light strip center lines corresponding to each third light-emitting area in the third target image; and, for each line structured light, determine a target third light strip center line corresponding to the line structured light from all the third light strip center lines, where the third target image includes N third light-emitting areas corresponding to the N line structured lights; For each first pixel point on the target first light strip center line, the determination module 62 projects the first pixel point onto the second target image to obtain a projected pixel point corresponding to the first pixel point, specifically: determining a third pixel point having the same pixel height as the first pixel point from the target third light strip centerline; determining a target 3D reconstruction point based on the first pixel point, the third pixel point and camera calibration parameters; configured to transform the target 3D reconstruction points into the projected pixel points based on a third calibration equation; A third calibration equation represents the functional relationship between pixel points in the second target image and the 3D reconstruction points.

[0133] Exemplarily, the camera calibration parameters include intrinsic camera parameters of a first camera, intrinsic camera parameters of a second camera, and extrinsic camera parameters between the first camera and the second camera; When determining the 3D points corresponding to the keypoint pairs based on the keypoint pairs and the camera calibration parameters, the determination module 62 specifically: performing distortion correction on the first pixel point using internal camera parameters of a first camera, and converting the pixel point after distortion correction into first homogeneous coordinates; performing distortion correction on the second pixel point using internal camera parameters of a second camera, and converting the pixel point after distortion correction into second homogeneous coordinates; The method is configured to determine a 3D point corresponding to the keypoint pair by triangulation based on the first homogeneous coordinates, the second homogeneous coordinates, the intrinsic camera parameters of the first camera, the intrinsic camera parameters of the second camera, and the extrinsic camera parameters.

[0134] Based on a similar idea to the above method, an embodiment of the present invention provides a three-dimensional imaging device including a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions executable by the processor, and the processor is configured to execute the image reconstruction method disclosed in the above embodiment of the present invention by executing the machine-executable instructions.

[0135] Based on a similar idea to the above method, an embodiment of the present invention further provides a machine-readable storage medium, in which several computer instructions are stored, and when the computer instructions are executed by a processor, the processor performs the image reconstruction method disclosed in the above embodiment of the present invention.

[0136] The machine-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, the machine-readable storage medium may be a RAM (Random Access Memory), a volatile memory, a non-volatile memory, a flash memory, a storage drive (e.g., a hard disk drive), a solid-state drive, any type of storage disk (e.g., an optical disk, a DVD, etc.), or a similar storage medium, or a combination thereof.

[0137] The systems, devices, modules, or units described in the above embodiments may be specifically realized by an entity or a product having some function. A typical realizing device is a computer, and the specific form of the computer may be a personal computer, a laptop computer, a mobile phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email sending / receiving device, a game console, a tablet PC, a wearable device, or any combination of these devices.

[0138] For convenience of description, the above-described device will be described as being divided into units according to their functions. Of course, when implementing the present invention, the functions of each unit can be realized by the same or multiple pieces of software and / or hardware.

[0139] Those skilled in the art will appreciate that embodiments of the present invention may be provided as a method, a system, or a computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product embodied in one or more computer-usable storage media (including, but not limited to, magnetic disk memory, CD-ROM, optical memory, etc.) having computer-usable program code thereon.

[0140] The present invention will be described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, where the instructions, executed by the processor of the general-purpose computer or other programmable data processing device, generate an apparatus for implementing the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0141] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specified manner, and the instructions stored in the computer-readable memory produce an article of manufacture that includes an instruction apparatus, which implements the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0142] These computer program instructions may be loaded into a computer or other programmable data processing device, and a series of operational steps may be executed on the computer or other programmable device to generate a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0143] The above description is merely an example of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included within the scope of the claims of the present invention.

Claims

1. 1. An image reconstruction method applied to a three-dimensional imaging device, the three-dimensional imaging device including a first camera, a second camera, and a multi-line laser, the method comprising: When the multi-line laser projects N lines of structured light onto the object to be measured, acquiring a first original image of the object to be measured collected by the first camera and acquiring a second original image of the object to be measured collected by the second camera, where N is a positive integer greater than 1; determining a first target image corresponding to the first original image and a second target image corresponding to the second original image, wherein the first target image includes N first light-emitting regions corresponding to the N line structured lights, and the second target image includes N second light-emitting regions corresponding to the N line structured lights; determining a first light strip centerline corresponding to each first light emitting area in the first target image and determining a second light strip centerline corresponding to each second light emitting area in the second target image; determining a plurality of keypoint pairs based on all of the first light strip centerlines and all of the second light strip centerlines, where for each keypoint pair, the keypoint pair includes a first pixel point on the first light strip centerline and a second pixel point on the second light strip centerline, the first pixel point and the second pixel point being pixel points corresponding to the same location on the object; determining a 3D point corresponding to the keypoint pair based on the keypoint pair and camera calibration parameters; generating a 3D reconstructed image of the object based on 3D points corresponding to the plurality of key point pairs; determining a plurality of keypoint pairs based on all of the first light strip centerlines and all of the second light strip centerlines; determining, for each line structured light, a target first light strip centerline and a target second light strip centerline corresponding to the line structured light from all of the first light strip centerlines and all of the second light strip centerlines; For each first pixel point on the target first light strip centerline, project the first pixel point onto the second target image to obtain a projected pixel point corresponding to the first pixel point, and select a second pixel point from the target second light strip centerline that corresponds to the projected pixel point; generating a keypoint pair based on the first pixel point and the second pixel point; 1. An image reconstruction method comprising:

2. Determining a first target image corresponding to the first original image and a second target image corresponding to the second original image includes: determining the first original image as the first target image and determining the second original image as the second target image; or performing binocular correction on the first original image and the second original image to obtain a first target image corresponding to the first original image and a second target image corresponding to the second original image; the binocular correction is used to make the same position point on the object to be measured have the same pixel height in the first target image and the second target image; 2. The method of claim 1 .

3. selecting a second pixel point from the target second light strip centerline that corresponds to the projected pixel point; determining a pixel point from the target second light strip centerline that has the same pixel height as the projected pixel point; If there is one determined pixel point, select the pixel point as the second pixel point; If the number of determined pixel points is at least two, determining a reprojection error between the at least two pixel points and the projected pixel point, and selecting a pixel point corresponding to a minimum reprojection error among the at least two pixel points as the second pixel point.

2. The method of claim 1 .

4. Projecting the first pixel point onto the second target image to obtain a projected pixel point corresponding to the first pixel point includes: obtaining a first calibration equation and a second calibration equation corresponding to the line structured light; transforming the first pixel points into target 3D reconstruction points based on the first calibration equation; transforming the target 3D reconstruction points into the projected pixel points based on the second calibration equation; the first calibration equation represents a functional relationship between pixel points in the first target image and three-dimensional reconstruction points, and the second calibration equation represents a functional relationship between pixel points in the second target image and three-dimensional reconstruction points; 2. The method of claim 1 .

5. The three-dimensional imaging device further includes a third camera, and the method further comprises: When the multi-line laser projects the N line structured light beams onto the object to be measured, acquiring a third original image of the object to be measured collected by the third camera, and determining a third target image corresponding to the third original image, wherein the third target image includes N third light-emitting areas corresponding to the N line structured light beams; determining a third light strip centerline corresponding to each third light-emitting region in the third target image; and determining, for each line structured light, a target third light strip center line corresponding to the line structured light from all third light strip center lines; For each first pixel point on the target first light strip centerline, projecting the first pixel point onto the second target image to obtain a projected pixel point corresponding to the first pixel point includes: determining a third pixel point having the same pixel height as the first pixel point from the target third light strip centerline; determining a target 3D reconstruction point based on the first pixel point, the third pixel point and camera calibration parameters; and transforming the target 3D reconstruction points into the projected pixel points based on a third calibration equation; a third calibration equation expressing a functional relationship between pixel points in the second target image and the three-dimensional reconstruction points; 2. The method of claim 1 .

6. the camera calibration parameters include intrinsic camera parameters of the first camera, intrinsic camera parameters of the second camera, and extrinsic camera parameters between the first camera and the second camera; determining 3D points corresponding to the keypoint pairs based on the keypoint pairs and camera calibration parameters, performing distortion correction on the first pixel point using internal camera parameters of the first camera, and converting the pixel point after distortion correction into first homogeneous coordinates; performing distortion correction on the second pixel point using internal camera parameters of the second camera, and converting the pixel point after distortion correction into second homogeneous coordinates; determining a 3D point corresponding to the keypoint pair by triangulation based on the first homogeneous coordinates, the second homogeneous coordinates, intrinsic camera parameters of the first camera, intrinsic camera parameters of the second camera, and the extrinsic camera parameters; 2. The method of claim 1 .

7. An image reconstruction apparatus to be applied to a three-dimensional imaging apparatus, the three-dimensional imaging apparatus including a first camera, a second camera, and a multi-line laser, the apparatus comprising: an acquisition module configured to acquire a first original image of the object to be measured collected by the first camera and acquire a second original image of the object to be measured collected by the second camera when the multi-line laser projects N lines of structured light onto the object to be measured, where N is a positive integer greater than 1; a determination module configured to: determine a first target image corresponding to the first original image and a second target image corresponding to the second original image; determine first light strip centerlines corresponding to each first light emitting area in the first target image; determine second light strip centerlines corresponding to each second light emitting area in the second target image; determine a plurality of key point pairs based on all the first light strip centerlines and all the second light strip centerlines; and determine three-dimensional points corresponding to the key point pairs based on the key point pairs and camera calibration parameters; a generation module configured to generate a 3D reconstructed image of the object based on 3D points corresponding to the plurality of key point pairs; the first target image includes N first light-emitting regions corresponding to the N line structured lights, and the second target image includes N second light-emitting regions corresponding to the N line structured lights; For each key point pair, the key point pair includes a first pixel point on the first light strip centerline and a second pixel point on the second light strip centerline, the first pixel point and the second pixel point corresponding to the same position on the object to be measured; When the determination module determines a plurality of keypoint pairs based on all of the first light strip centerlines and all of the second light strip centerlines, For each line structured light, determine a target first light strip centerline and a target second light strip centerline corresponding to the line structured light from all of the first light strip centerlines and all of the second light strip centerlines; For each first pixel point on the target first light strip centerline, project the first pixel point onto the second target image to obtain a projected pixel point corresponding to the first pixel point; and select a second pixel point from the target second light strip centerline corresponding to the projected pixel point; configured to generate a keypoint pair based on the first pixel point and the second pixel point; An image reconstruction device characterized by:

8. 1. A three-dimensional imaging device including a processor and a machine-readable storage medium, The machine-readable storage medium stores machine-executable instructions that are executable by the processor; The processor is configured to execute the machine-executable instructions to perform the method of any one of claims 1 to 6. A three-dimensional imaging device.

Images