3D reconstruction method and laser scanner

Abstract

The present application discloses a three-dimensional reconstruction method and a laser scanner, which belong to the field of laser scanning. The three-dimensional reconstruction method includes: constructing three-dimensional point cloud data corresponding to a single-frame image frame based on the image frame of the object to be reconstructed collected by the binocular camera in the laser scanner; the three-dimensional point cloud data includes multiple reconstruction points and the three-dimensional coordinates of each reconstruction point; the reconstruction point is a spatial coordinate point obtained by three-dimensional reconstruction of the pixel points in the image frame; interpolation processing is performed on at least part of the reconstruction points in the three-dimensional point cloud data to obtain the three-dimensional coordinates of the target interpolation point. The three-dimensional reconstruction method of the present application can obtain point cloud data that far exceeds the camera resolution, increase the number of points in a single-frame image frame, and thus improve the scanning efficiency; and the three-dimensional coordinates corresponding to the reconstructed target interpolation point conform to the surface change trend of the object to be reconstructed, and have high sampling precision and accuracy, thereby improving the user experience.

Description

Technical Field

[0001] The present application belongs to the field of laser scanning, and in particular relates to a three-dimensional reconstruction method and a laser scanner. Background Art

[0002] Binocular structured light 3D reconstruction is a technology that uses two cameras and a structured light projector to obtain the 3D shape and depth information of an object's surface. It is widely used in industrial inspection, intelligent manufacturing, autonomous driving, robot navigation, virtual reality, security monitoring and other fields.

[0003] In related technologies, binocular reconstruction is based on the points on the pixels of the image; that is, the number of points in a single frame is related to the image pixels, one pixel reconstructs a three-dimensional coordinate point, and then subsequent reconstruction and other operations are performed based on the reconstructed three-dimensional coordinate point; the reconstructed point cloud data distribution can include: 1) reconstructing a surface based on the reconstruction method of the entire surface radiation matching such as phase and speckle; 2) reconstructing a curve based on reconstruction methods such as laser lines and projection lines.

[0004] However, the number of points output under the current reconstruction method is limited by the camera resolution, and the maximum number of reconstruction points is fixed, resulting in the inability to increase the scanning speed, affecting the reconstruction effect and thus the user experience. Summary of the Invention

[0005] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a three-dimensional reconstruction method and a laser scanner to increase the number of points in a single image frame, thereby improving scanning efficiency.

[0006] In a first aspect, the present application provides a three-dimensional reconstruction method, the method comprising:

[0007] Based on image frames of the object to be reconstructed captured by a binocular camera in a laser scanner, three-dimensional point cloud data corresponding to a single image frame is constructed; the three-dimensional point cloud data includes a plurality of reconstruction points and the three-dimensional coordinates of each of the reconstruction points; the reconstruction points are spatial coordinate points obtained by three-dimensionally reconstructing pixel points in the image frame;

[0008] Interpolation processing is performed on at least part of the reconstructed points in the three-dimensional point cloud data to obtain three-dimensional coordinates of target interpolation points.

[0009] According to the 3D reconstruction method of the present application, by interpolating the 3D point cloud data obtained based on a single-frame image frame, the target interpolation point and the 3D coordinates of the target interpolation point are obtained, so that the 3D coordinates of the new position on the object to be reconstructed other than the reconstruction point can be obtained, so that the output point amount is not limited by the resolution of the binocular camera, and point cloud data far exceeding the camera resolution is obtained, thereby increasing the output point amount of the single-frame image frame and thus improving the scanning efficiency; and the 3D coordinates corresponding to the reconstructed target interpolation points are consistent with the surface change trend of the object to be reconstructed, and have high sampling precision and accuracy, thereby improving the user experience.

[0010] According to one embodiment of the present application, interpolating at least some of the reconstructed points in the three-dimensional point cloud data to obtain three-dimensional coordinates of target interpolation points includes:

[0011] fitting at least part of the reconstructed points in the three-dimensional point cloud data based on the three-dimensional point cloud data to obtain a fitting object;

[0012] Interpolation processing is performed on at least part of the reconstructed points in the fitting object to obtain the three-dimensional coordinates of the target interpolation points.

[0013] According to one embodiment of the present application, fitting at least some of the reconstructed points in the three-dimensional point cloud data based on the three-dimensional point cloud data to obtain a fitting object includes:

[0014] Surface fitting is performed on at least part of the reconstructed points based on the three-dimensional point cloud data to obtain a target surface.

[0015] According to one embodiment of the present application, performing interpolation processing on at least some of the reconstructed points in the fitting object to obtain the three-dimensional coordinates of the target interpolation point includes:

[0016] Performing interpolation processing on at least part of the reconstructed points in the target curved surface to obtain first coordinates of the target interpolation points;

[0017] The first coordinates are transformed to obtain the three-dimensional coordinates of the target interpolation point.

[0018] According to one embodiment of the present application, performing interpolation processing on at least some of the reconstructed points in the target surface to obtain first coordinates of the target interpolation points includes:

[0019] Obtaining second coordinates of at least part of the reconstructed points in the three-dimensional point cloud data on the target surface;

[0020] Interpolation processing is performed on the second coordinate to obtain the target interpolation point and the first coordinate of the target interpolation point on the target surface.

[0021] According to one embodiment of the present application, transforming the first coordinate to obtain the three-dimensional coordinates of the target interpolation point includes:

[0022] Using a three-dimensional polynomial fitting algorithm to perform surface fitting on the three-dimensional coordinates of some reconstructed points in the three-dimensional point cloud data to obtain a preset fitting formula;

[0023] The first coordinate is processed based on the preset fitting formula to obtain the three-dimensional coordinates of the target interpolation point.

[0024] According to one embodiment of the present application, the three-dimensional polynomial fitting algorithm is used to perform surface fitting on the three-dimensional coordinates of some reconstructed points in the three-dimensional point cloud data to obtain a preset fitting formula, including:

[0025] Converting the projection coordinates of the reconstructed point on the target surface into spatial coordinates based on the preset fitting formula;

[0026] Solving the optimal value of a first coefficient of the preset fitting formula based on the three-dimensional point cloud data and the spatial coordinates to obtain a value of the first coefficient;

[0027] The preset fitting formula is determined based on the value of the first coefficient.

[0028] According to one embodiment of the present application, fitting at least some of the reconstructed points in the three-dimensional point cloud data based on the three-dimensional point cloud data to obtain a fitting object includes:

[0029] Curve fitting is performed on at least part of the reconstructed points based on the three-dimensional point cloud data to obtain a target curve.

[0030] According to one embodiment of the present application, performing interpolation processing on at least some of the reconstructed points in the fitting object to obtain the three-dimensional coordinates of the target interpolation point includes:

[0031] performing interpolation processing on at least some of the reconstructed points in the target curve to obtain first coordinates of the target interpolation points;

[0032] The first coordinates are transformed to obtain the three-dimensional coordinates of the target interpolation point.

[0033] According to one embodiment of the present application, performing interpolation processing on at least some of the reconstructed points in the target curve to obtain first coordinates of the target interpolation points includes:

[0034] Obtaining third coordinates of at least some of the reconstructed points in the three-dimensional point cloud data in a target direction;

[0035] Interpolation processing is performed on the third coordinate to obtain the target interpolation point and the first coordinate of the target interpolation point in the target direction.

[0036] According to one embodiment of the present application, transforming the first coordinate to obtain the three-dimensional coordinates of the target interpolation point includes:

[0037] Using a two-dimensional polynomial fitting algorithm, curve fitting is performed on the three-dimensional coordinates of some reconstructed points in the three-dimensional point cloud data to obtain a preset fitting formula;

[0038] The first coordinate is processed based on the preset fitting formula to obtain the three-dimensional coordinates of the target interpolation point.

[0039] According to one embodiment of the present application, the two-dimensional polynomial fitting algorithm is used to perform curve fitting on the three-dimensional coordinates of some reconstructed points in the three-dimensional point cloud data to obtain a preset fitting formula, including:

[0040] Converting the projection coordinates of the reconstructed point in the target direction into spatial coordinates in the target direction based on the preset fitting formula;

[0041] Solving the optimal value of the second coefficient of the preset fitting formula based on the three-dimensional point cloud data and the spatial coordinates to obtain a value of the second coefficient;

[0042] Based on the value of the second coefficient, a preset fitting formula corresponding to the target direction is determined.

[0043] According to one embodiment of the present application, performing interpolation processing on at least some of the reconstructed points in the fitting object to obtain the three-dimensional coordinates of the target interpolation point includes:

[0044] performing interpolation processing on at least part of the reconstructed points in the fitting object to obtain first coordinates of the target interpolation point;

[0045] The first coordinate is processed based on a preset fitting formula to obtain the three-dimensional coordinates of the target interpolation point; the preset fitting formula is determined by fitting the three-dimensional point cloud data.

[0046] According to one embodiment of the present application, interpolating at least some of the reconstructed points in the three-dimensional point cloud data to obtain three-dimensional coordinates of target interpolation points includes:

[0047] A linear interpolation algorithm is used to interpolate at least some of the reconstructed points in the three-dimensional point cloud data to obtain the three-dimensional coordinates of the target interpolation point.

[0048] According to one embodiment of the present application, interpolating at least some of the reconstructed points in the three-dimensional point cloud data to obtain three-dimensional coordinates of target interpolation points includes:

[0049] A nearest neighbor interpolation algorithm is used to interpolate at least some of the reconstructed points based on the grayscale values ​​of at least some of the reconstructed points in the three-dimensional point cloud data to obtain the three-dimensional coordinates of the target interpolation point and the grayscale value of the target interpolation point.

[0050] According to one embodiment of the present application, interpolating at least some of the reconstructed points in the three-dimensional point cloud data to obtain three-dimensional coordinates of target interpolation points includes:

[0051] Based on the sampling accuracy, determine the target number of interpolation points;

[0052] The three-dimensional point cloud data is interpolated based on the target number of interpolation points to obtain the target number of target interpolation points and the three-dimensional coordinates of each target interpolation point.

[0053] In a second aspect, the present application provides a three-dimensional reconstruction method, the method comprising:

[0054] According to the three-dimensional reconstruction method as described in the first aspect, the three-dimensional coordinates of at least one target interpolation point are obtained:

[0055] The object to be reconstructed is three-dimensionally reconstructed based on the three-dimensional coordinates of the target interpolation point and the three-dimensional point cloud data.

[0056] According to the 3D reconstruction method of the present application, by interpolating the three-dimensional point cloud data obtained based on a single-frame image frame, the coordinates in the world coordinate system are obtained, so that the three-dimensional coordinates of the new positions on the object to be reconstructed other than the reconstruction points can be obtained, so that the output point quantity is not limited by the resolution of the binocular camera, and point cloud data far exceeding the camera resolution is obtained, thereby increasing the output point quantity of the single-frame image frame and thus improving the scanning efficiency; and the three-dimensional coordinates corresponding to the reconstructed target interpolation points are consistent with the surface change trend of the object to be reconstructed, with high sampling precision and accuracy, thereby improving the accuracy of the reconstructed three-dimensional model and improving the scanning effect.

[0057] In a third aspect, the present application provides a three-dimensional reconstruction device, comprising:

[0058] A first processing module is configured to construct three-dimensional point cloud data corresponding to a single image frame based on image frames of the object to be reconstructed captured by a binocular camera in a laser scanner; the three-dimensional point cloud data includes a plurality of reconstruction points and the three-dimensional coordinates of each of the reconstruction points; the reconstruction points are spatial coordinate points obtained by three-dimensionally reconstructing pixel points in the image frame;

[0059] The second processing module is used to perform interpolation processing on at least part of the reconstructed points in the three-dimensional point cloud data to obtain the three-dimensional coordinates of the target interpolation points.

[0060] According to the 3D reconstruction device of the present application, by interpolating the 3D point cloud data constructed based on a single-frame image frame, the target interpolation point and the 3D coordinates of the target interpolation point are obtained, so that the 3D coordinates of the new position on the object to be reconstructed other than the reconstruction point can be obtained, so that the output point amount is not limited by the resolution of the binocular camera, and point cloud data far exceeding the camera resolution is obtained, thereby increasing the output point amount of the single-frame image frame, thereby improving the scanning efficiency; and the 3D coordinates corresponding to the reconstructed target interpolation points are consistent with the surface change trend of the object to be reconstructed, with high sampling precision and accuracy, thereby improving the user experience.

[0061] In a fourth aspect, the present application provides a three-dimensional reconstruction device, comprising:

[0062] A third processing module is configured to obtain the three-dimensional coordinates of at least one target interpolation point according to the three-dimensional reconstruction method described in the first aspect:

[0063] A fourth processing module is configured to perform three-dimensional reconstruction on the object to be reconstructed based on the three-dimensional coordinates of the target interpolation point and the three-dimensional point cloud data.

[0064] In a fifth aspect, the present application provides a laser scanner, comprising:

[0065] A binocular camera, a laser group and a control device, wherein the control device is used to execute the three-dimensional reconstruction method as described in the first aspect or the three-dimensional reconstruction method as described in the second aspect.

[0066] In a sixth aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and runnable on the processor, wherein when the processor executes the computer program, it implements the three-dimensional reconstruction method as described in the first aspect or the three-dimensional reconstruction method as described in the second aspect.

[0067] In the seventh aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the three-dimensional reconstruction method as described in the first aspect or the three-dimensional reconstruction method as described in the second aspect.

[0068] In an eighth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the three-dimensional reconstruction method as described in the first aspect or the three-dimensional reconstruction method as described in the second aspect.

[0069] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0070] By interpolating the three-dimensional point cloud data constructed based on a single-frame image frame, the target interpolation point and its three-dimensional coordinates are obtained, so that the three-dimensional coordinates of the new position on the object to be reconstructed other than the reconstruction point can be obtained, so that the output point quantity is not limited by the resolution of the binocular camera, and point cloud data far exceeding the camera resolution is obtained, which increases the output point quantity of a single-frame image frame and thus improves the scanning efficiency; and the three-dimensional coordinates corresponding to the reconstructed target interpolation point are consistent with the surface change trend of the object to be reconstructed, with high sampling precision and accuracy, thereby improving the user experience.

[0071] Furthermore, by fitting the three-dimensional point cloud data lines constructed based on a single-frame image frame and performing interpolation processing on the obtained fitting object to obtain the three-dimensional coordinates of the target interpolation point on the object to be reconstructed, the interpolation accuracy can be improved, so that the target interpolation point obtained by upsampling is more in line with the surface change trend of the object to be reconstructed, and is suitable for three-dimensional reconstruction scenarios of irregular objects and complex structure objects; it can also reduce single-point noise fluctuations, improve interpolation accuracy, further improve the sampling effect, and has high universality.

[0072] Furthermore, the reconstructed three-dimensional point cloud data is interpolated by a direct linear interpolation algorithm to obtain new interpolation points, which can increase the number of points. The operation is simple, convenient and easy to implement.

[0073] Furthermore, by performing surface fitting on the reconstructed three-dimensional point cloud data and performing interpolation processing based on the fitted target surface, a surface that is more consistent with the surface change law of the object to be reconstructed can be obtained, thereby improving the precision and accuracy of the interpolation points obtained by interpolation and improving the sampling effect.

[0074] Furthermore, by performing curve fitting on the reconstructed three-dimensional point cloud data and performing interpolation processing based on the fitted target curve, the operation is simple, fast and easy to implement, which can improve sampling efficiency and thus improve scanning efficiency.

[0075] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become obvious from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0077] Figure 1 This is one of the flow charts of the three-dimensional reconstruction method provided in the embodiment of the present application;

[0078] Figure 2 This is one of the intermediate result schematic diagrams of the three-dimensional reconstruction method provided in the embodiment of the present application;

[0079] Figure 3 This is the second schematic diagram of the intermediate result of the three-dimensional reconstruction method provided in the embodiment of the present application;

[0080] Figure 4 This is one of the structural diagrams of the three-dimensional reconstruction device provided in the embodiments of the present application;

[0081] Figure 5 This is the second flow chart of the three-dimensional reconstruction method provided in the embodiment of the present application;

[0082] Figure 6 is a schematic diagram of the results of the three-dimensional reconstruction method provided in an embodiment of the present application;

[0083] Figure 7 This is the second structural diagram of the three-dimensional reconstruction device provided in an embodiment of the present application;

[0084] Figure 8 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0085] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0086] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0087] The following describes in detail the 3D reconstruction method, 3D reconstruction device, electronic device, and readable storage medium provided by the embodiments of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0088] The three-dimensional reconstruction method may be applied to a terminal, and may be specifically executed by hardware or software in the terminal.

[0089] The three-dimensional reconstruction method provided in the embodiments of the present application can be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the three-dimensional reconstruction method. The electronic devices mentioned in the embodiments of the present application include but are not limited to mobile phones, tablets, computers, cameras, and wearable devices. The three-dimensional reconstruction method provided in the embodiments of the present application is described below using an electronic device as an example of the execution subject.

[0090] A handheld laser scanner is a high-precision 3D measurement device that measures the surface features of an object by emitting a laser beam and receiving the reflected light. Handheld laser scanners are used in industrial design, architectural surveying, archaeological excavations, and other fields to obtain 3D models of objects.

[0091] In related technologies, when acquiring a three-dimensional model, the image frames are mainly collected through the binocular camera inside the handheld laser scanner, and the binocular reconstruction technology is used to construct three-dimensional point cloud data based on the image frames, wherein the binocular reconstruction is based on the points on the pixels of the image; that is, the number of points in a single frame is related to the image pixels, and one pixel reconstructs a three-dimensional coordinate point, and then subsequent reconstruction and other operations are performed based on the reconstructed three-dimensional coordinate point; the reconstructed point cloud data distribution may include: 1) Reconstructing a surface based on the reconstruction method of the entire surface radiation matching such as phase and speckle; 2) Reconstructing a curve based on the reconstruction method such as laser line and projection line. However, the number of points reconstructed by the above method is limited by the camera resolution, and the maximum number of reconstructed points is fixed, resulting in the inability to increase the scanning speed and affecting the scanning efficiency; in addition, since the single-pixel reconstruction is unrelated to each other, the single-point noise fluctuates greatly, affecting the reconstruction precision and accuracy, thereby affecting the user experience.

[0092] To solve at least one of the above technical problems, an embodiment of the present application provides a three-dimensional reconstruction method, a three-dimensional reconstruction method, a three-dimensional reconstruction device, a three-dimensional reconstruction device, an electronic device, and a readable storage medium. Below, in conjunction with the accompanying drawings, the three-dimensional reconstruction method, the three-dimensional reconstruction method, the three-dimensional reconstruction device, the three-dimensional reconstruction device, the electronic device, and the readable storage medium provided in the embodiment of the present application are described in detail through specific embodiments and their application scenarios.

[0093] like Figure 1 As shown, the three-dimensional reconstruction method includes: step 110 and step 120.

[0094] Step 110: constructing three-dimensional point cloud data corresponding to a single image frame based on the image frames of the object to be reconstructed captured by the binocular camera in the laser scanner; the three-dimensional point cloud data includes multiple reconstruction points and the three-dimensional coordinates of each reconstruction point; the reconstruction point is a spatial coordinate point obtained by three-dimensionally reconstructing the pixel points in the image frame;

[0095] In this step, a laser scanner is a high-precision measurement device that can include components such as a laser and a binocular camera. The laser emits a laser beam that reflects upon the surface of the scanned object. The binocular camera captures these reflected laser beams and, using binocular reconstruction technology, calculates the three-dimensional coordinate information of the object's surface. A laser scanner can include a single laser that emits a single laser beam, or multiple lasers that emit intersecting laser beams. These laser beams intersect in space, forming a cross-shaped pattern that captures three-dimensional information about the object's surface, thereby improving scanning accuracy and efficiency.

[0096] The binocular camera includes two cameras, each used to capture reflections from lasers emitted by each laser group at the object to be reconstructed, to produce two image frames. Using binocular reconstruction technology, any point on the object to be reconstructed is imaged by the two cameras separately, obtaining the pixel coordinates of the corresponding pixel in the two image frames. Based on the parameter matrices of the two cameras, a coordinate transformation relationship is then established to calculate the world coordinates of the point. By performing similar processing on multiple pixels in the two image frames, the world coordinates corresponding to multiple points on the object to be reconstructed (i.e., reconstruction points) can be calculated, thereby generating three-dimensional point cloud data. In some embodiments, the binocular camera can be two cameras with identical internal parameters, arranged in parallel with their optical axes.

[0097] In some embodiments, before step 110, constructing three-dimensional point cloud data corresponding to a single image frame based on the image frames of the object to be reconstructed captured by the binocular camera in the laser scanner, the method may further include:

[0098] Calibrate the binocular camera.

[0099] In this embodiment, before image frame acquisition, the external parameters of the dual-target cameras should be calibrated to reduce errors caused by different internal parameters of the binocular cameras and different shooting states.

[0100] Step 120: Perform interpolation processing on at least some of the reconstructed points in the three-dimensional point cloud data to obtain three-dimensional coordinates of target interpolation points.

[0101] In this step, the number of output points corresponding to the three-dimensional point cloud data constructed by the binocular reconstruction algorithm is determined based on the resolution of the binocular camera. One pixel point in the image frame can correspond to the calculated three-dimensional coordinates of a reconstructed point; that is, the maximum value of the number of output points (equivalent to the maximum number of reconstructed points included in the three-dimensional point cloud data) is the number of pixels included in the image frame captured by the binocular camera.

[0102] At least part of the reconstruction points includes two reconstruction points or more than two reconstruction points.

[0103] The target interpolation point is a new point obtained by interpolating the three-dimensional point cloud data, which is different from the reconstructed point obtained by directly mapping the pixel points in the image frame. The target interpolation point may include one or more interpolation points. By interpolating at least part of the reconstructed points to obtain the interpolation points, the output point quantity can be increased.

[0104] In some embodiments, step 120 of interpolating at least some of the reconstructed points in the three-dimensional point cloud data to obtain the three-dimensional coordinates of the target interpolation points may include:

[0105] A linear interpolation algorithm is used to interpolate at least some of the reconstructed points in the three-dimensional point cloud data to obtain the three-dimensional coordinates of the target interpolation points.

[0106] In this embodiment, after reconstructing the three-dimensional point cloud data, the target interpolation point and the three-dimensional coordinates corresponding to the target interpolation point can be directly interpolated through a linear interpolation algorithm, such as by reconstructing the pixel x and y directions and directly performing linear interpolation to obtain a new interpolation point.

[0107] Of course, in other embodiments, other interpolation algorithms may also be used, such as cubic spline interpolation or Kriging interpolation algorithm, etc., which are not described in detail in this application.

[0108] According to the three-dimensional reconstruction method provided in the embodiment of the present application, the reconstructed three-dimensional point cloud data is interpolated by a direct linear interpolation algorithm to obtain new interpolation points, which can increase the number of points. The operation is simple, convenient and easy to implement.

[0109] In some embodiments, step 120 of interpolating at least some of the reconstructed points in the three-dimensional point cloud data to obtain the three-dimensional coordinates of the target interpolation points may include:

[0110] A nearest neighbor interpolation algorithm is used to interpolate at least some of the reconstructed points based on the grayscale values ​​of at least some of the reconstructed points in the three-dimensional point cloud data to obtain the three-dimensional coordinates of the target interpolation points and the grayscale values ​​of the target interpolation points.

[0111] In this embodiment, after obtaining the target interpolation point, the grayscale value of the reconstructed point closest to the target interpolation point can be assigned to the target interpolation point using a nearest neighbor interpolation algorithm, thereby obtaining the grayscale value corresponding to the new interpolation point.

[0112] Of course, in other embodiments, the average value of the grayscale values ​​of the reconstructed points near the target interpolation point may also be determined as the grayscale value of the target interpolation point, which is not limited in this application.

[0113] According to the 3D reconstruction method provided in the embodiment of the present application, by interpolating the 3D point cloud data obtained based on a single-frame image frame, the target interpolation point and the 3D coordinates of the target interpolation point are obtained, so that the 3D coordinates of new positions on the object to be reconstructed other than the reconstruction point can be obtained, so that the output point quantity is not limited by the resolution of the binocular camera, and point cloud data far exceeding the camera resolution is obtained, thereby increasing the output point quantity of the single-frame image frame, thereby improving the scanning efficiency; and the 3D coordinates corresponding to the reconstructed target interpolation points are consistent with the surface change trend of the object to be reconstructed, with high sampling precision and accuracy, thereby improving the user experience.

[0114] In some embodiments, step 120 of interpolating at least some of the reconstructed points in the three-dimensional point cloud data to obtain the three-dimensional coordinates of the target interpolation points may include:

[0115] fitting at least a portion of the reconstructed points in the three-dimensional point cloud data based on the three-dimensional point cloud data to obtain a fitting object;

[0116] Interpolation processing is performed on at least some of the reconstructed points in the fitting object to obtain three-dimensional coordinates of target interpolation points.

[0117] In this embodiment, before interpolation, a line or surface fitting can be performed based on some of the reconstructed points in the reconstructed 3D point cloud data to obtain a fitting object. Interpolation is then performed on the fitted object to obtain the target interpolation point and its 3D coordinates. The fitting object can include a fitted plane, a fitted surface, a fitted line, or a fitted curve. Linear interpolation or other interpolation algorithms can also be used during the interpolation of the fitted object.

[0118] According to the 3D reconstruction method provided in the embodiment of the present application, by fitting the 3D point cloud data lines constructed based on a single-frame image frame, and performing interpolation processing on the obtained fitting object to obtain the 3D coordinates of the target interpolation point on the object to be reconstructed, the interpolation accuracy can be improved, so that the target interpolation point obtained by upsampling is more in line with the surface change trend of the object to be reconstructed, and it is suitable for 3D reconstruction scenarios of irregular objects and complex structure objects; it can also reduce single-point noise fluctuations, improve interpolation accuracy, further improve the sampling effect, and has high universality.

[0119] In some embodiments, interpolating at least some of the reconstructed points in the fitting object to obtain the three-dimensional coordinates of the target interpolation point may include:

[0120] Performing interpolation processing on at least some of the reconstructed points in the fitting object to obtain first coordinates of target interpolation points;

[0121] The first coordinate is processed based on a preset fitting formula to obtain the three-dimensional coordinates of the target interpolation point.

[0122] In this embodiment, the preset fitting formula is a formula used to characterize the surface change trend of the object to be reconstructed.

[0123] The preset fitting formula is determined by fitting the three-dimensional point cloud data.

[0124] The preset fitting formula may include, but is not limited to, a cubic polynomial, a quartic polynomial, or a quintic polynomial. In the process of solving the polynomial, a least squares method or other extreme value solving algorithm may be used to solve the polynomial to obtain a preset fitting formula with a better fitting effect, thereby obtaining a fitting object that better conforms to the surface change trend of the object to be reconstructed.

[0125] According to the three-dimensional reconstruction method provided in the embodiment of the present application, polynomial fitting is performed, and the operation is simple and convenient.

[0126] The following describes the embodiments of the present application in detail from two different perspectives of fitting objects.

[0127] 1. The fitting object is a surface

[0128] In some embodiments, fitting at least some of the reconstructed points in the three-dimensional point cloud data based on the three-dimensional point cloud data to obtain a fitting object may include:

[0129] Surface fitting is performed on at least part of the reconstructed points based on the three-dimensional point cloud data to obtain a target surface.

[0130] In this embodiment, surface fitting can be performed based on the three-dimensional coordinates corresponding to some adjacent reconstruction points in the three-dimensional point cloud data to obtain the target surface, such as by reconstructing the x-direction and y-direction of the pixels. Figure 2 As shown, one grid point corresponds to one reconstruction point, and the grid point stores the three-dimensional coordinates of the reconstruction point corresponding to the grid point. Figure 2 The reconstructed points within a certain range are fitted to obtain a high-order surface, and the high-order surface is used as the target surface to obtain the target interpolation point by upsampling on the high-order surface.

[0131] In some embodiments, interpolating at least some of the reconstructed points in the fitting object to obtain the three-dimensional coordinates of the target interpolation point may include:

[0132] Performing interpolation processing on at least some of the reconstructed points in the target surface to obtain first coordinates of the target interpolation points;

[0133] The first coordinate is transformed to obtain the three-dimensional coordinate of the target interpolation point.

[0134] In this embodiment, the first coordinate may be coordinates in any two directions of the three-dimensional coordinates, such as an x ​​coordinate and a y coordinate.

[0135] After obtaining the first coordinate, the z coordinate can be determined based on the x coordinate and the y coordinate using the fitting parameters of the target surface, thereby obtaining the three-dimensional coordinates of the target interpolation point on the object to be reconstructed.

[0136] In some embodiments, performing interpolation processing on at least some of the reconstructed points in the target curved surface to obtain first coordinates of the target interpolation points may include:

[0137] Obtaining second coordinates of at least some of the reconstructed points in the three-dimensional point cloud data on the target surface;

[0138] The second coordinate is interpolated to obtain a target interpolation point and a first coordinate of the target interpolation point on the target surface.

[0139] In this embodiment, the target surface may be an orthographic projection surface formed by the x-axis direction and the y-axis direction, so as to ensure that the coordinates (x, y) on the projection surface are the same as the (x, y) in the three-dimensional space coordinates.

[0140] The second coordinate is the projection coordinate obtained by projecting the three-dimensional coordinate of the reconstructed point onto the target surface. The second coordinate is the coordinate in the same dimension as the first coordinate. For example, the second coordinate can be the coordinate of the reconstructed point in the x-axis direction and the y-axis direction.

[0141] Continue to refer to Figure 2 The high-order surface shown in the figure can be interpolated to obtain the fitted surface. The nearest neighbor points of multiple reconstruction points can be found and linear interpolation can be performed to obtain new points. Taking the reconstruction points A (x, y), B (x+1, y), C (x, y+1) and D (x+1, y+1) as examples, the target interpolation point G can be obtained by interpolating each reconstruction point:

[0142] Pi=(P<x,y> +P<x+1,y> +P<x+1,y+1> +P<x,y+1> ) / 4;

[0143] Where Pi is the first coordinate of the target interpolation point G.

[0144] In some embodiments, two adjacent reconstruction points can be selected in the fitted high-order surface, and linear interpolation is performed on the two adjacent reconstruction points to obtain a new point. Figure 2 , taking the reconstruction point A (x, y) and the reconstruction point C (x, y+1) as an example, the target interpolation point I can be obtained by interpolating the reconstruction point A and the reconstruction point B:

[0145] Pi=(P<x,y> +P<x,y+1> ) / 2;

[0146] Wherein, Pi is the first coordinate of the target interpolation point I.

[0147] In some embodiments, transforming the first coordinate to obtain the three-dimensional coordinates of the target interpolation point may include:

[0148] A three-dimensional polynomial fitting algorithm is used to perform surface fitting on the three-dimensional coordinates of some reconstructed points in the three-dimensional point cloud data to obtain a preset fitting formula;

[0149] The first coordinate is processed based on a preset fitting formula to obtain the three-dimensional coordinates of the target interpolation point.

[0150] In this embodiment, a three-dimensional polynomial fitting algorithm is used to fit a high-order surface, and a preset fitting formula is used to characterize the correlation between coordinate values ​​in any two dimensions of the three-dimensional coordinates and the coordinate value in another dimension.

[0151] During the fitting process, based on the known three-dimensional point cloud data, a cubic, quartic, quintic or other degree polynomial can be selected and fitted using the least squares method to obtain a fitting polynomial with better fitting effect as the preset fitting formula.

[0152] In some embodiments, a three-dimensional polynomial fitting algorithm is used to perform surface fitting on the three-dimensional coordinates of some reconstructed points in the three-dimensional point cloud data to obtain a preset fitting formula, which may include:

[0153] The projection coordinates of the reconstructed points on the target surface are converted into spatial coordinates based on a preset fitting formula;

[0154] Solve the optimal value of the first coefficient of the preset fitting formula based on the three-dimensional point cloud data and the spatial coordinates to obtain the value of the first coefficient;

[0155] Based on the value of the first coefficient, a preset fitting formula is determined.

[0156] In this embodiment, the surface equation (i.e., the preset fitting formula) is: h(u,v)=au 2 +buv+cv 2 +du+ev+f

[0157] Among them, a, b, c, d, e and f are the first coefficients used to describe the fitting coefficients of the fitting surface; (ui, vi, hi) is the three-dimensional coordinate of the i-th reconstruction point, (ui, vi, hi)∈C, i=1, 2,..., n, n is a positive integer, and C is the centroid set.

[0158] List the linear equations as:

[0159]

[0160] The above linear equations can be simplified to: AX=h;

[0161] Solve using the linear least squares method, that is, , the first coefficient can be obtained, thereby obtaining the surface equation, that is, obtaining a preset fitting formula for characterizing the correlation relationship between the projection coordinates on the target surface and the coordinate value of another dimension in the three-dimensional coordinates.

[0162] Substituting the first coordinate (u, v) corresponding to the target interpolation point into the preset fitting formula, the corresponding h coordinate can be obtained, thereby obtaining the three-dimensional coordinates of the target interpolation point.

[0163] According to the three-dimensional reconstruction method provided in the embodiment of the present application, by performing surface fitting on the reconstructed three-dimensional point cloud data and performing interpolation processing based on the fitted target surface, a surface that is more in line with the surface change law of the object to be reconstructed can be obtained, thereby improving the precision and accuracy of the interpolation points obtained by interpolation and improving the sampling effect.

[0164] 2. The fitting object is a curve

[0165] In some embodiments, fitting at least some of the reconstructed points in the three-dimensional point cloud data based on the three-dimensional point cloud data to obtain a fitting object may include:

[0166] Curve fitting is performed on at least part of the reconstructed points based on the three-dimensional point cloud data to obtain a target curve.

[0167] In this embodiment, reconstruction points within a certain range may be first extracted to perform polynomial fitting to obtain a fitting curve, and then the fitting curve may be upsampled to obtain new target interpolation points.

[0168] Figure 3 An example of a target curve is shown.

[0169] In some embodiments, interpolating at least some of the reconstructed points in the fitting object to obtain the three-dimensional coordinates of the target interpolation point may include:

[0170] Performing interpolation processing on at least some of the reconstructed points in the target curve to obtain first coordinates of the target interpolation points;

[0171] The first coordinate is transformed to obtain the three-dimensional coordinate of the target interpolation point.

[0172] In this embodiment, the first coordinate may be a projection coordinate in the x-axis direction, a projection coordinate in the y-axis direction, or a projection coordinate in the z-axis direction.

[0173] Continue to refer to Figure 3 The target curve shown in the figure takes the reconstruction point E(t) and the reconstruction point F(t+1) as an example. The target interpolation point H can be obtained by interpolating the reconstruction point E and the reconstruction point F:

[0174] Pi=(P <t>+P<t+1> ) / 2;

[0175] Wherein, Pi is the first coordinate of the target interpolation point H.

[0176] After obtaining the first coordinates, the first coordinates are transformed based on the fitting parameters to obtain the three-dimensional coordinates of the target interpolation point on the object to be reconstructed.

[0177] In some embodiments, interpolating at least some of the reconstructed points in the target curve to obtain first coordinates of the target interpolation points may include:

[0178] Obtaining third coordinates of at least some of the reconstructed points in the three-dimensional point cloud data in the target direction;

[0179] The third coordinate is interpolated to obtain a target interpolation point and a first coordinate of the target interpolation point in a target direction.

[0180] In this embodiment, the third coordinate corresponds one-to-one to the dimension corresponding to the first coordinate.

[0181] The target direction may include an x-direction, a y-direction, and a z-direction.

[0182] During the actual execution process, the reconstructed points in the three-dimensional point cloud data should be projected along the x-direction, y-direction and z-direction respectively to obtain the third coordinates in each target direction; then, the same projection points should be interpolated in each target direction to obtain the first coordinates of the target interpolation points in different target directions.

[0183] In some embodiments, transforming the first coordinate to obtain the three-dimensional coordinates of the target interpolation point may include:

[0184] Using a two-dimensional polynomial fitting algorithm, curve fitting is performed on the three-dimensional coordinates of some reconstructed points in the three-dimensional point cloud data to obtain a preset fitting formula;

[0185] The first coordinate is processed based on a preset fitting formula to obtain the three-dimensional coordinates of the target interpolation point.

[0186] In this embodiment, a two-dimensional polynomial fitting algorithm is used to fit the curve, and a preset fitting formula is used to characterize the correlation between the projection coordinates in the target direction and the three-dimensional coordinates.

[0187] During the fitting process, based on the known three-dimensional point cloud data, a cubic, quartic, quintic or other degree polynomial can be selected and fitted using the least squares method to obtain a fitting polynomial with better fitting effect as the preset fitting formula.

[0188] In some embodiments, a two-dimensional polynomial fitting algorithm is used to perform curve fitting on the three-dimensional coordinates of some reconstructed points in the three-dimensional point cloud data to obtain a preset fitting formula, which may include:

[0189] The projection coordinates of the reconstructed points in the target direction are converted into spatial coordinates in the target direction based on a preset fitting formula;

[0190] Solve the optimal value of the second coefficient of the preset fitting formula based on the three-dimensional point cloud data and the spatial coordinates to obtain the value of the second coefficient;

[0191] Based on the value of the second coefficient, a preset fitting formula corresponding to the target direction is determined.

[0192] In this embodiment, the target direction may include an x-axis direction, a y-axis direction, and a z-axis direction.

[0193] Each direction corresponds to a preset fitting formula.

[0194] Assume that the curve equation (i.e. the preset fitting formula) is:

[0195] x=f(t);

[0196] y=g(t);

[0197] z=h(t);

[0198] Where t is the projection coordinate in the target direction.

[0199] In the actual fitting process, you can customize the polynomial to fit. Taking cubic polynomial fitting as an example, the curve equation can be obtained as follows:

[0200] x(t)=a0+a1*t+a2*t 2 +a3*t 3 ;

[0201] y(t)=b0+b1*t+b2*t 2 +b3*t 3 ;

[0202] z(t)=c0+c1*t+c2*t 2 +c3*t 3 ;

[0203] Wherein, a0, a1, a2, a3, b0, b1, b2, b3, c0, c1, c2 and c3 are the second coefficients used to describe the fitting coefficients of the fitting curve, ti is the projection coordinate of the i-th reconstruction point in the target direction, i = 1, 2, ..., n, where n is a positive integer.

[0204] Substitute the known three-dimensional point cloud data into the curve equation and use the least squares method or other methods to solve the second coefficient to obtain an optimal polynomial fitting curve, that is, obtain the preset fitting formula; then substitute the first coordinate t corresponding to the target interpolation point in each target direction into the preset fitting formula corresponding to the target direction to obtain the three-dimensional coordinates of the target interpolation point.

[0205] According to the three-dimensional reconstruction method provided in the embodiment of the present application, curve fitting is performed on the reconstructed three-dimensional point cloud data, and interpolation processing is performed based on the fitted target curve. The operation is simple, fast and easy to implement, which can improve sampling efficiency and thus improve scanning efficiency.

[0206] This application provides a variety of different interpolation methods, which can be flexibly selected based on actual needs.

[0207] In some embodiments, step 120 of interpolating at least some of the reconstructed points in the three-dimensional point cloud data to obtain the three-dimensional coordinates of the target interpolation points may include:

[0208] Based on the sampling accuracy, determine the target number of interpolation points;

[0209] The three-dimensional point cloud data is interpolated based on the target number of interpolation points to obtain the target number of target interpolation points and the three-dimensional coordinates of each target interpolation point.

[0210] In this embodiment, the target number is the number of outgoing points that need to be increased.

[0211] In some embodiments, the number of targets may be user-defined, or may be determined based on sampling or scanning accuracy.

[0212] The number of outgoing points to be added may be selected according to actual needs, and an interpolation process may be performed according to the number of outgoing points to be added to obtain a corresponding number of interpolation points and the first coordinates corresponding to the interpolation points.

[0213] According to the three-dimensional reconstruction method provided in the embodiment of the present application, a corresponding number of interpolation points can be added according to the number of output points required in the actual application process, so that the number of output points is not limited by the resolution of the binocular camera, and point cloud data far exceeding the camera resolution is obtained, thereby improving scanning efficiency.

[0214] The 3D reconstruction method provided in the embodiment of the present application can be executed by a 3D reconstruction device. In the embodiment of the present application, the 3D reconstruction device provided in the embodiment of the present application is described by taking the 3D reconstruction method performed by the 3D reconstruction device as an example.

[0215] An embodiment of the present application also provides a three-dimensional reconstruction device.

[0216] like Figure 4 As shown, the three-dimensional reconstruction device includes: a first processing module 410 and a second processing module 420 .

[0217] The first processing module 410 is configured to construct three-dimensional point cloud data corresponding to a single image frame based on image frames of the object to be reconstructed captured by the binocular camera in the laser scanner. The three-dimensional point cloud data includes a plurality of reconstruction points and the three-dimensional coordinates of each reconstruction point. The reconstruction points are spatial coordinate points obtained by three-dimensionally reconstructing pixels in the image frame.

[0218] The second processing module 420 is configured to perform interpolation processing on at least some of the reconstructed points in the three-dimensional point cloud data to obtain three-dimensional coordinates of target interpolation points.

[0219] According to the 3D reconstruction device provided by the embodiment of the present application, by interpolating the 3D point cloud data constructed based on a single-frame image frame, the target interpolation point and the 3D coordinates of the target interpolation point are obtained, so that the 3D coordinates of new positions on the object to be reconstructed other than the reconstruction point can be obtained, so that the output point amount is not limited by the resolution of the binocular camera, and point cloud data far exceeding the camera resolution is obtained, thereby increasing the output point amount of the single-frame image frame, thereby improving the scanning efficiency; and the 3D coordinates corresponding to the reconstructed target interpolation points are consistent with the surface change trend of the object to be reconstructed, with high sampling precision and accuracy, thereby improving the user experience.

[0220] In some embodiments, the second processing module 420 may also be used to:

[0221] fitting at least a portion of the reconstructed points in the three-dimensional point cloud data based on the three-dimensional point cloud data to obtain a fitting object;

[0222] Interpolation processing is performed on at least some of the reconstructed points in the fitting object to obtain three-dimensional coordinates of target interpolation points.

[0223] In some embodiments, the second processing module 420 may also be used to:

[0224] Surface fitting is performed on at least part of the reconstructed points based on the three-dimensional point cloud data to obtain a target surface.

[0225] In some embodiments, the second processing module 420 may also be used to:

[0226] Performing interpolation processing on at least some of the reconstructed points in the target surface to obtain first coordinates of the target interpolation points;

[0227] The first coordinate is transformed to obtain the three-dimensional coordinate of the target interpolation point.

[0228] In some embodiments, the second processing module 420 may also be used to:

[0229] Obtaining second coordinates of at least some of the reconstructed points in the three-dimensional point cloud data on the target surface;

[0230] The second coordinate is interpolated to obtain a target interpolation point and a first coordinate of the target interpolation point on the target surface.

[0231] In some embodiments, the second processing module 420 may also be used to:

[0232] A three-dimensional polynomial fitting algorithm is used to perform surface fitting on the three-dimensional coordinates of some reconstructed points in the three-dimensional point cloud data to obtain a preset fitting formula;

[0233] The first coordinate is processed based on a preset fitting formula to obtain the three-dimensional coordinates of the target interpolation point.

[0234] In some embodiments, the second processing module 420 may also be used to:

[0235] The projection coordinates of the reconstructed points on the target surface are converted into spatial coordinates based on a preset fitting formula;

[0236] Solve the optimal value of the first coefficient of the preset fitting formula based on the three-dimensional point cloud data and the spatial coordinates to obtain the value of the first coefficient;

[0237] Based on the value of the first coefficient, a preset fitting formula is determined.

[0238] In some embodiments, the second processing module 420 may also be used to:

[0239] Curve fitting is performed on at least part of the reconstructed points based on the three-dimensional point cloud data to obtain a target curve.

[0240] In some embodiments, the second processing module 420 may also be used to:

[0241] Performing interpolation processing on at least some of the reconstructed points in the target curve to obtain first coordinates of the target interpolation points;

[0242] The first coordinate is transformed to obtain the three-dimensional coordinate of the target interpolation point.

[0243] In some embodiments, the second processing module 420 may also be used to:

[0244] Obtaining third coordinates of at least some of the reconstructed points in the three-dimensional point cloud data in the target direction;

[0245] The third coordinate is interpolated to obtain a target interpolation point and a first coordinate of the target interpolation point in a target direction.

[0246] In some embodiments, the second processing module 420 may also be used to:

[0247] Using a two-dimensional polynomial fitting algorithm, curve fitting is performed on the three-dimensional coordinates of some reconstructed points in the three-dimensional point cloud data to obtain a preset fitting formula;

[0248] The first coordinate is processed based on a preset fitting formula to obtain the three-dimensional coordinates of the target interpolation point.

[0249] In some embodiments, the second processing module 420 may also be used to:

[0250] The projection coordinates of the reconstructed points in the target direction are converted into spatial coordinates in the target direction based on a preset fitting formula;

[0251] Solve the optimal value of the second coefficient of the preset fitting formula based on the three-dimensional point cloud data and the spatial coordinates to obtain the value of the second coefficient;

[0252] Based on the value of the second coefficient, a preset fitting formula corresponding to the target direction is determined.

[0253] In some embodiments, the second processing module 420 may also be used to:

[0254] Performing interpolation processing on at least some of the reconstructed points in the fitting object to obtain first coordinates of target interpolation points;

[0255] The first coordinate is processed based on a preset fitting formula to obtain the three-dimensional coordinates of the target interpolation point; the preset fitting formula is determined by fitting the three-dimensional point cloud data.

[0256] In some embodiments, the second processing module 420 may also be used to:

[0257] A linear interpolation algorithm is used to interpolate at least some of the reconstructed points in the three-dimensional point cloud data to obtain the three-dimensional coordinates of the target interpolation points.

[0258] In some embodiments, the second processing module 420 may also be used to:

[0259] A nearest neighbor interpolation algorithm is used to interpolate at least some of the reconstructed points based on the grayscale values ​​of at least some of the reconstructed points in the three-dimensional point cloud data to obtain the three-dimensional coordinates of the target interpolation points and the grayscale values ​​of the target interpolation points.

[0260] In some embodiments, the second processing module 420 may also be used to:

[0261] Based on the sampling accuracy, determine the target number of interpolation points;

[0262] The three-dimensional point cloud data is interpolated based on the target number of interpolation points to obtain the target number of target interpolation points and the three-dimensional coordinates of each target interpolation point.

[0263] The three-dimensional reconstruction device in the embodiments of the present application can be an electronic device or a component of an electronic device, such as an integrated circuit or chip. The electronic device can be a terminal or other device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA). It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc., and the embodiments of the present application do not specifically limit this.

[0264] The 3D reconstruction device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0265] The three-dimensional reconstruction device provided in the embodiment of the present application can achieve Figures 1 to 3 To avoid repetition, the various processes implemented in the method embodiment are not described here.

[0266] The embodiment of the present application also provides a three-dimensional reconstruction method.

[0267] like Figure 5 As shown, the three-dimensional reconstruction method includes: step 510 and step 520.

[0268] Step 510: Obtain the three-dimensional coordinates of at least one target interpolation point according to the three-dimensional reconstruction method described in any of the above embodiments:

[0269] In this step, each image frame captured by the binocular camera can be upsampled based on a similar method to obtain one or more target interpolation points and the three-dimensional coordinates of the target interpolation points.

[0270] Step 520: Perform three-dimensional reconstruction on the object to be reconstructed based on the three-dimensional coordinates of the target interpolation point and the three-dimensional point cloud data.

[0271] In this step, by shooting multiple image frames, such as collecting multiple image frames of the object to be reconstructed at different angles, the three-dimensional point cloud data corresponding to each image frame and the target interpolation points obtained by interpolation based on the three-dimensional point cloud data and the three-dimensional coordinates of the target interpolation points are calculated. By combining all the three-dimensional coordinates, a three-dimensional model of the object to be reconstructed can be reconstructed, such as Figure 6 shown.

[0272] Of course, in other embodiments, after obtaining the three-dimensional coordinates of the target interpolation point on the object to be reconstructed, data fusion, filtering and other scanning and measurement work can be performed based on the three-dimensional coordinates of the target interpolation point on the object to be reconstructed and the three-dimensional point cloud data.

[0273] According to the 3D reconstruction method provided in the embodiment of the present application, by interpolating the 3D point cloud data constructed based on a single-frame image frame, the coordinates in the world coordinate system are obtained, so that the 3D coordinates of the new positions on the object to be reconstructed other than the reconstruction points can be obtained, so that the output point quantity is not limited by the resolution of the binocular camera, and point cloud data far exceeding the camera resolution is obtained, thereby increasing the output point quantity of the single-frame image frame and improving the scanning efficiency; and the 3D coordinates corresponding to the reconstructed target interpolation points are consistent with the surface change trend of the object to be reconstructed, with high sampling precision and accuracy, thereby improving the accuracy of the reconstructed 3D model and improving the scanning effect.

[0274] The 3D reconstruction method provided in the embodiment of the present application can be executed by a 3D reconstruction device. In the embodiment of the present application, the 3D reconstruction device provided in the embodiment of the present application is described by taking the 3D reconstruction method performed by the 3D reconstruction device as an example.

[0275] An embodiment of the present application also provides a three-dimensional reconstruction device.

[0276] like Figure 7 As shown, the three-dimensional reconstruction device includes: a fourth processing module 710 and a fifth processing module 720 .

[0277] The fourth processing module 710 is configured to obtain the three-dimensional coordinates of at least one target interpolation point according to the three-dimensional reconstruction method described in any of the above embodiments:

[0278] The fifth processing module 720 is configured to perform three-dimensional reconstruction on the object to be reconstructed based on the three-dimensional coordinates of the target interpolation point on the object to be reconstructed and the three-dimensional point cloud data.

[0279] According to the 3D reconstruction device provided by the embodiment of the present application, by interpolating the 3D point cloud data constructed based on a single-frame image frame, the coordinates in the world coordinate system are obtained, so that the 3D coordinates of the new positions on the object to be reconstructed other than the reconstruction points can be obtained, so that the output point quantity is not limited by the resolution of the binocular camera, and point cloud data far exceeding the camera resolution is obtained, thereby increasing the output point quantity of the single-frame image frame and thus improving the scanning efficiency; and the 3D coordinates corresponding to the reconstructed target interpolation points are consistent with the surface change trend of the object to be reconstructed, with high sampling precision and accuracy, thereby improving the accuracy of the reconstructed 3D model and improving the scanning effect.

[0280] The three-dimensional reconstruction device in the embodiments of the present application can be an electronic device or a component of an electronic device, such as an integrated circuit or chip. The electronic device can be a terminal or other device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA). It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc., and the embodiments of the present application do not specifically limit this.

[0281] The 3D reconstruction device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0282] The three-dimensional reconstruction device provided in the embodiment of the present application can achieve Figure 5 and Figure 6 To avoid repetition, the various processes implemented in the method embodiment are not described here.

[0283] An embodiment of the present application also provides a laser scanner.

[0284] The laser scanner includes a binocular camera, a laser group and a control device.

[0285] In this embodiment, the laser group includes a plurality of lasers emitting cross-beam lasers.

[0286] The binocular camera is electrically connected to the control device, and the control device is used to execute the three-dimensional reconstruction method described in any of the above embodiments or the three-dimensional reconstruction method described in any of the above embodiments.

[0287] The binocular camera includes two cameras, which are used to respectively capture the reflections generated by the lasers emitted by each laser of the laser group to the object to be reconstructed, so as to obtain two image frames.

[0288] A control device is configured to employ binocular reconstruction technology to obtain pixel coordinates of pixels corresponding to any point on an object to be reconstructed by imaging the two cameras separately, and then calculate the world coordinates of the point by establishing a coordinate transformation relationship based on the parameter matrices of the two cameras. Similarly processing is performed on multiple pixel points in the two image frames to calculate the world coordinates corresponding to multiple reconstructed points, thereby obtaining three-dimensional point cloud data. Based on the obtained three-dimensional point cloud data, the three-dimensional reconstruction method described in any of the above embodiments is executed for upsampling to obtain one or more interpolation points and the three-dimensional coordinates of the interpolation points.

[0289] The obtained reconstruction points and interpolation points are used to represent the three-dimensional coordinate points of the object to be reconstructed in the three-dimensional space.

[0290] In some embodiments, the control device is also used to perform three-dimensional reconstruction of the object to be reconstructed based on the three-dimensional coordinates of the interpolation points and the three-dimensional point cloud data after obtaining one or more interpolation points and the three-dimensional coordinates of the interpolation points, so as to obtain a three-dimensional model of the object to be reconstructed, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0291] The controller may be the electronic device mentioned above.

[0292] According to the laser scanner provided in the embodiment of the present application, by interpolating the three-dimensional point cloud data constructed based on a single-frame image frame to obtain the coordinates of the interpolation point in the world coordinate system, the three-dimensional coordinates of the new position can be obtained in addition to the reconstructed points obtained by direct mapping based on the pixel points, so that the output point quantity is not limited by the resolution of the binocular camera, and point cloud data far exceeding the camera resolution is obtained, thereby increasing the output point quantity of the single-frame image frame and thus improving the scanning efficiency; and the three-dimensional coordinates corresponding to the reconstructed target interpolation point are consistent with the surface change trend of the object to be reconstructed, with high sampling precision and accuracy, thereby improving the user experience.

[0293] In some embodiments, as Figure 8 As shown, an embodiment of the present application also provides an electronic device 800, including a processor 801, a memory 802, and a computer program stored in the memory 802 and executable on the processor 801. When the program is executed by the processor 801, the above-mentioned three-dimensional reconstruction method or each process of the three-dimensional reconstruction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0294] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0295] An embodiment of the present application also provides a non-transitory computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the various processes of the above-mentioned three-dimensional reconstruction method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0296] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0297] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned three-dimensional reconstruction method when executed by a processor.

[0298] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0299] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned three-dimensional reconstruction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0300] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0301] It should be noted that, in this document, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, but may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0302] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of this application.

[0303] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

[0304] Throughout this specification, references to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "example," "specific example," or "some examples" indicate that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present application. In this specification, illustrative uses of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0305] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and intent of the present application, and that the scope of the present application is defined by the claims and their equivalents.< / t>

Claims

1. A three-dimensional reconstruction method, characterized in that: include: Based on the image frames of the object to be reconstructed captured by the binocular camera in the laser scanner, three-dimensional point cloud data corresponding to the single image frame is constructed; The three-dimensional point cloud data includes a plurality of reconstruction points and the three-dimensional coordinates of each of the reconstruction points; the reconstruction points are spatial coordinate points obtained by three-dimensionally reconstructing the pixel points in the image frame; performing interpolation processing on at least some of the reconstructed points in the three-dimensional point cloud data to obtain three-dimensional coordinates of target interpolation points; The interpolation processing of at least part of the reconstructed points in the three-dimensional point cloud data to obtain the three-dimensional coordinates of the target interpolation point includes: fitting at least part of the reconstructed points in the three-dimensional point cloud data based on the three-dimensional point cloud data to obtain a fitting object; Performing interpolation processing on at least some of the reconstructed points in the fitting object to obtain the three-dimensional coordinates of the target interpolation point; the fitting object includes a target surface or a target curve; When the fitting object is the target curved surface, performing surface fitting on at least part of the reconstructed points based on the three-dimensional point cloud data to obtain the target curved surface; Performing interpolation processing on at least part of the reconstructed points in the target curved surface to obtain first coordinates of the target interpolation points; Converting the projection coordinates of the reconstructed point on the target surface into spatial coordinates based on the preset fitting formula; Solving the optimal value of a first coefficient of the preset fitting formula based on the three-dimensional point cloud data and the spatial coordinates to obtain a value of the first coefficient; Determining the preset fitting formula based on the value of the first coefficient; Processing the first coordinate based on the preset fitting formula to obtain the three-dimensional coordinates of the target interpolation point; When the fitting object is the target curve, performing curve fitting on at least part of the reconstructed points based on the three-dimensional point cloud data to obtain a target curve; performing interpolation processing on at least some of the reconstructed points in the target curve to obtain first coordinates of the target interpolation points; Converting the projection coordinates of the reconstructed point in the target direction into spatial coordinates in the target direction based on the preset fitting formula; Solving the optimal value of the second coefficient of the preset fitting formula based on the three-dimensional point cloud data and the spatial coordinates to obtain a value of the second coefficient; Determining a preset fitting formula corresponding to the target direction based on the value of the second coefficient; The first coordinate is processed based on the preset fitting formula to obtain the three-dimensional coordinates of the target interpolation point.

2. The three-dimensional reconstruction method according to claim 1, characterized in that: The interpolation processing is performed on at least part of the reconstructed points in the target curved surface to obtain the first coordinates of the target interpolation points, including: Obtaining second coordinates of at least part of the reconstructed points in the three-dimensional point cloud data on the target surface; Interpolation processing is performed on the second coordinate to obtain the target interpolation point and the first coordinate of the target interpolation point on the target surface.

3. The three-dimensional reconstruction method according to claim 1, wherein: The interpolation processing is performed on at least part of the reconstructed points in the target curve to obtain the first coordinates of the target interpolation points, including: Obtaining third coordinates of at least some of the reconstructed points in the three-dimensional point cloud data in a target direction; Interpolation processing is performed on the third coordinate to obtain the target interpolation point and the first coordinate of the target interpolation point in the target direction.

4. The three-dimensional reconstruction method according to claim 1, characterized in that: The interpolation processing of at least part of the reconstructed points in the three-dimensional point cloud data to obtain the three-dimensional coordinates of the target interpolation point includes: A linear interpolation algorithm is used to interpolate at least some of the reconstructed points in the three-dimensional point cloud data to obtain the three-dimensional coordinates of the target interpolation point.

5. The three-dimensional reconstruction method according to any one of claims 1 to 4, characterized in that: The interpolation processing of at least part of the reconstructed points in the three-dimensional point cloud data to obtain the three-dimensional coordinates of the target interpolation point includes: A nearest neighbor interpolation algorithm is used to interpolate at least some of the reconstructed points based on the grayscale values ​​of at least some of the reconstructed points in the three-dimensional point cloud data to obtain the three-dimensional coordinates of the target interpolation point and the grayscale value of the target interpolation point.

6. The three-dimensional reconstruction method according to any one of claims 1 to 4, characterized in that: The interpolation processing of at least part of the reconstructed points in the three-dimensional point cloud data to obtain the three-dimensional coordinates of the target interpolation point includes: Based on the sampling accuracy, determine the target number of interpolation points; The three-dimensional point cloud data is interpolated based on the target number of interpolation points to obtain the target number of target interpolation points and the three-dimensional coordinates of each target interpolation point.

7. A three-dimensional reconstruction device, characterized in that: include: A first processing module is configured to construct three-dimensional point cloud data corresponding to a single image frame based on image frames of the object to be reconstructed acquired by a binocular camera in a laser scanner; The three-dimensional point cloud data includes a plurality of reconstruction points and the three-dimensional coordinates of each of the reconstruction points; the reconstruction points are spatial coordinate points obtained by three-dimensionally reconstructing the pixel points in the image frame; a second processing module, configured to perform interpolation processing on at least some of the reconstructed points in the three-dimensional point cloud data to obtain three-dimensional coordinates of target interpolation points; The second processing module is configured to fit at least part of the reconstructed points in the three-dimensional point cloud data based on the three-dimensional point cloud data to obtain a fitting object; Performing interpolation processing on at least some of the reconstructed points in the fitting object to obtain three-dimensional coordinates of the target interpolation points; The fitting object includes a target surface or a target curve; The second processing module is configured to: When the fitting object is the target curved surface, performing surface fitting on at least part of the reconstructed points based on the three-dimensional point cloud data to obtain the target curved surface; Performing interpolation processing on at least part of the reconstructed points in the target curved surface to obtain first coordinates of the target interpolation points; Converting the projection coordinates of the reconstructed point on the target surface into spatial coordinates based on the preset fitting formula; Solving the optimal value of a first coefficient of the preset fitting formula based on the three-dimensional point cloud data and the spatial coordinates to obtain a value of the first coefficient; Determining the preset fitting formula based on the value of the first coefficient; Processing the first coordinate based on the preset fitting formula to obtain the three-dimensional coordinates of the target interpolation point; When the fitting object is the target curve, performing curve fitting on at least part of the reconstructed points based on the three-dimensional point cloud data to obtain a target curve; performing interpolation processing on at least some of the reconstructed points in the target curve to obtain first coordinates of the target interpolation points; Converting the projection coordinates of the reconstructed point in the target direction into spatial coordinates in the target direction based on the preset fitting formula; Solving the optimal value of the second coefficient of the preset fitting formula based on the three-dimensional point cloud data and the spatial coordinates to obtain a value of the second coefficient; Determining a preset fitting formula corresponding to the target direction based on the value of the second coefficient; The first coordinate is processed based on the preset fitting formula to obtain the three-dimensional coordinates of the target interpolation point.

8. A laser scanner, characterized in that: include: A binocular camera, a laser group and a control device, wherein the control device is used to execute the three-dimensional reconstruction method according to any one of claims 1 to 6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the three-dimensional reconstruction method according to any one of claims 1 to 6 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the three-dimensional reconstruction method according to any one of claims 1 to 6 is implemented.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the three-dimensional reconstruction method according to any one of claims 1 to 6 is implemented.

Images