A multi-line laser based global depth estimation method, system, medium, and program product

Abstract

This application relates to the field of laser projection 3D reconstruction technology, and discloses a global depth estimation method, system, medium, and program product based on multi-line lasers. The laser projection 3D reconstruction system includes a camera and four laser light sources. The camera is located at the center of a quadrilateral formed by the laser light sources. The laser light sources in the horizontal direction emit lasers of the same wavelength, while those in the vertical direction emit lasers of different wavelengths. The method includes: performing laser separation on the laser projection image acquired by the camera to obtain two image matrices; processing each image matrix based on the camera's intrinsic parameter matrix and the calibrated laser curve mapping relationship to obtain a contour point cloud; performing point cloud completion on the image matrix containing the contour point cloud using a visual basic model to obtain a depth map; and converting the depth map into an image containing a globally dense point cloud. This application enables the acquisition of reconstructed images with high frame rate and high accuracy of the global field of view.

Description

Technical Field

[0001] This invention relates to the field of laser projection 3D reconstruction technology, and in particular to a global depth estimation method, system, medium, and program product based on multi-line lasers. Background Technology

[0002] Laser projection 3D reconstruction involves projecting coded laser light onto the scene under test using a laser light source. The coded laser light is modulated by the scene under test, and a 2D camera captures an image of the scene containing the laser lines. The scene image is then processed to obtain a 3D point cloud of the projection area.

[0003] In the exemplary technologies, laser projection methods include dot matrix, line array, and area array encoding, but none of them can simultaneously guarantee high frame rate, high accuracy, and global field of view reconstruction. Summary of the Invention

[0004] This invention provides a global depth estimation method and related equipment based on multi-line lasers to solve problems that are difficult to solve in the prior art.

[0005] Firstly, this application provides a global depth estimation method based on multi-line lasers, applied to a laser projection 3D reconstruction system. The laser projection 3D reconstruction system includes a camera and four laser light sources. The camera is located at the center of a quadrilateral formed by the laser light sources. The laser light sources positioned horizontally emit lasers of the same wavelength, while the laser light sources positioned vertically emit lasers of different wavelengths. The global depth estimation method based on multi-line lasers includes:

[0006] Acquire the laser projection image captured by the camera, the laser projection image being formed by lasers emitted from each of the laser light sources; The laser projection image is separated into two image matrices. Based on the camera's intrinsic parameter matrix and the calibration laser curve mapping relationship, each image matrix is ​​processed to obtain the contour point cloud of the laser emitted by the laser source in each image matrix. The image matrix containing the contour point cloud is filled with point cloud by using a visual basic model to obtain the corresponding depth map of the laser projection image, and the depth map is converted into an image containing a globally dense point cloud.

[0007] In some implementations, processing each image matrix based on the camera's intrinsic parameter matrix and the calibrated laser curve mapping relationship to obtain the contour point cloud of the laser emitted by the laser source in each image matrix includes: Each image matrix is ​​divided into four equal parts to obtain multiple sub-images corresponding to the image matrix; Based on the camera's intrinsic parameter matrix and the calibration laser curve mapping relationship, each sub-image is processed to obtain four contour point clouds corresponding to each image matrix.

[0008] In some implementations, dividing each of the image matrices into four equal parts includes: The image matrix is ​​divided equally in the horizontal direction and equally in the vertical direction.

[0009] In some embodiments, before acquiring the laser projection image captured by the camera, the method further includes: A set of images of a chessboard in different poses is obtained. The set of images includes a first calibration image and a second calibration image. The first calibration image is an image of the chessboard without laser projection, and the second calibration image is an image of the chessboard including laser projection. Based on the multiple sets of images, the laser projection 3D reconstruction system is calibrated to obtain the camera's intrinsic parameter matrix and the calibration laser curve mapping relationship.

[0010] In some implementations, the step of performing point cloud completion on the image matrix containing the contour point cloud using a visual base model includes: The contour point cloud is adjusted to the same resolution as the camera to obtain an intermediate image; The intermediate image is input into the visual base model, and the intermediate image is filled in with point cloud data through the visual base model to obtain the corresponding depth map of the laser projection image.

[0011] In some implementations, the visual base model performs feature extraction on the intermediate image at multiple stages and constructs target image features based on the image features extracted at each stage; the image features extracted by the visual base model at the final stage of the intermediate image are added to the target image features to obtain the depth map.

[0012] In some embodiments, the laser source arranged in the horizontal direction emits a first laser, and the laser source arranged in the vertical direction emits a second laser, wherein the first laser and the second laser are different; or, the laser source arranged in the vertical direction emits a first laser, and the laser source arranged in the horizontal direction emits a second laser.

[0013] Secondly, this application also provides a laser projection 3D reconstruction system, including a camera and four laser light sources. The camera is located at the center of a quadrilateral formed by the laser light sources. The laser light sources arranged horizontally emit laser light of the same wavelength, and the laser light sources arranged vertically emit laser light of different wavelengths. The laser projection 3D reconstruction system also includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method described in the first aspect.

[0014] Thirdly, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0015] Fourthly, this application also provides a computer program product, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0016] The present invention has the following beneficial effects: The laser projection 3D reconstruction system includes four laser light sources. The camera is located at the center of the quadrilateral formed by the laser light sources. The laser light sources set in the horizontal direction emit laser light of the same wavelength, and the laser light sources set in the vertical direction emit laser light of different wavelengths. The system acquires the laser projection image collected by the camera, and performs laser separation on the laser projection image to obtain two image matrices. Through the camera's intrinsic parameter matrix and the calibration of the laser curve mapping relationship, each image matrix is ​​processed to obtain the contour point cloud of the laser emitted by the laser light source in the image matrix. Then, the image matrix containing the contour point cloud is filled in using the visual basic model to obtain the depth map corresponding to the laser projection image. The depth map is then converted into an image containing a globally dense point cloud, thereby obtaining a reconstructed image with high frame rate and high precision of global field of view.

[0017] In addition to the objectives, features and advantages described above, the present invention has other objectives, features and advantages.

[0018] The present invention will now be described in further detail with reference to the figures. Attached Figure Description

[0019] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 These are schematic diagrams of the laser projection three-dimensional reconstruction system involved in the embodiments of this application. (a) is a schematic diagram of the laser projection from below, and (b) is a schematic diagram of the laser projection from above.

[0020] Figure 2 This is a flowchart illustrating the global depth estimation method based on multi-line laser provided in this application embodiment.

[0021] Figure 3 This is a schematic diagram of the point cloud completion process of the laser projection image involved in the embodiments of this application. (a) is a laser projection image containing laser lines and contour point clouds, (b) is a global depth map, and (c) is an image containing a global dense point cloud. Detailed Implementation

[0022] The technical solution of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms "an" or "a," and similar terms, do not indicate a quantity limitation, but rather indicate the presence of at least one.

[0024] Laser projection 3D reconstruction involves projecting coded laser light onto the scene under test using a laser light source. The coded laser light is modulated by the scene under test, and a 2D camera captures an image of the scene containing the laser lines. The scene image is then processed to obtain a 3D point cloud of the projection area.

[0025] In the exemplary technologies, laser projection methods include dot matrix, line array, and area array encoding, but none of them can simultaneously guarantee high frame rate, high accuracy, and global field of view reconstruction.

[0026] To address the aforementioned issues, this application provides a global depth estimation method and related equipment based on multi-line lasers. The laser projection 3D reconstruction system includes four laser light sources. A camera is located at the center of a quadrilateral formed by the laser light sources. The horizontally positioned laser light sources emit lasers of the same wavelength, while the vertically positioned laser light sources emit lasers of different wavelengths. The system acquires laser projection images captured by the camera and performs laser separation on the laser projection images to obtain two image matrices. Using the camera's intrinsic parameter matrix and the calibrated laser curve mapping relationship, each image matrix is ​​processed to obtain the contour point cloud of the laser emitted by the laser light source in the image matrix. Then, the image matrix containing the contour point cloud is filled in using a visual basic model to obtain the depth map corresponding to the laser projection image. Finally, the depth map is converted into an image containing a globally dense point cloud, thereby obtaining a reconstructed image with high frame rate and high accuracy of the global field of view.

[0027] The laser projection 3D reconstruction system involved in this application will be described in detail below.

[0028] The laser projection 3D reconstruction system includes a camera and four laser light sources arranged in a quadrilateral shape, with the camera positioned at the center. The system also includes a quadrilateral shield with elongated slits of the same size on each side and a circular hole at its center. Each laser light source projects its laser beam through its corresponding elongated slit, and the camera captures the laser projection images formed by the laser beams emitted from each light source through the circular hole. (See attached image for details.) Figure 1 The laser projection diagram shown in (a) is viewed from below and... Figure 1 The diagram in (b) shows a top-view view of the laser projection. The camera can be a general-purpose high-precision industrial camera.

[0029] The following combination Figure 1 This application provides a detailed description of the global depth estimation method based on multi-line lasers.

[0030] Reference Figure 2 , Figure 2 The flowchart of the multi-line laser global depth estimation method provided in this application is as follows: Step S201: Acquire the laser projection image captured by the camera. The laser projection image is formed by lasers emitted from various laser sources.

[0031] In this embodiment, the executing entity is the laser projection 3D reconstruction system. For ease of description, the term "system" will be used to refer to the laser projection 3D reconstruction system below.

[0032] The system controls multiple laser sources to emit lasers, so that each laser source can project a laser of a specific wavelength at a specific construction angle to form a laser projection. The camera in the system captures the image of the laser projection to obtain the laser projection image. The system then acquires the laser projection image captured by the camera. In other words, the laser projection image is formed by the lasers emitted by each laser source.

[0033] In this embodiment, the system is illustrated using four laser light sources. The laser lines projected by light sources positioned opposite each other are parallel in the scene, while adjacent light sources intersect at 90° angles. The projection direction of all light sources is slightly deflected in the opposite direction to the line connecting the center of the light source and the center of the camera, i.e., projected away from the central camera direction. Furthermore, parallel laser light sources use the same wavelength, while adjacent laser light sources use different wavelengths. That is, each laser light source positioned horizontally emits a first laser, and each laser light source positioned vertically emits a second laser. For example, the first laser emitted by the two horizontal laser sources is a 405nm blue laser, and the second laser emitted by the two vertical laser sources is a 650nm red laser. Alternatively, each laser light source positioned horizontally emits a second laser, and each laser light source positioned vertically emits a first laser. For example, the second laser emitted by the two horizontal laser sources is a 650nm red laser and a 405nm blue laser, and the first laser emitted by the two vertical laser sources is a 405nm blue laser. With this system configuration, accurate separation of laser lines can be achieved in a single frame image.

[0034] Step S202: Perform laser separation on the laser projection image to obtain two image matrices. Based on the camera's intrinsic parameter matrix and the calibration laser curve mapping relationship, process each image matrix to obtain the contour point cloud of the laser emitted by the laser source in each image matrix.

[0035] After obtaining the laser projection image, laser separation is performed on the laser projection image to obtain two image matrices. For example, the camera's three channels separate the 405nm blue light projection and the 650nm red light projection to obtain two image matrices. The two image matrices are images containing parallel laser lines of two laser lines. For example, one image matrix contains two parallel blue laser lines, and the other image matrix contains two parallel red laser lines.

[0036] After obtaining the image matrix, each image matrix is ​​processed based on the camera's intrinsic parameter matrix and the calibration laser curve mapping relationship to obtain the contour point cloud of the laser emitted by the laser source in each image matrix.

[0037] In one example, the image matrix is ​​divided into multiple sub-images. The division can be random. Then, based on the camera's intrinsic parameter matrix and the calibrated laser curve mapping relationship, each sub-image is processed to obtain the contour point cloud corresponding to each image matrix. The intrinsic parameter matrix is ​​constructed based on the camera's focal length and the position of the light spot, and can be determined through calibration.

[0038] In another example, the image matrix is ​​divided into four equal parts to obtain multiple sub-images. Then, based on the camera's intrinsic parameter matrix and the calibration laser curve mapping relationship, each sub-image is processed to obtain four contour point clouds corresponding to each image matrix. Furthermore, the image matrix can be divided equally in the horizontal direction and equally in the vertical direction to obtain four sub-images of the image matrix.

[0039] For example, in an image matrix, since the four laser sources in the system project towards the direction away from the central camera, each individual laser line, after scene modulation, is located in the corresponding half of the image region. For instance, for two vertically parallel laser lines in the image matrix, the laser line on the left will still be in the left half of the matrix, and vice versa. Therefore, the image matrix can be equally divided into left and right parts and top and bottom parts, and the corresponding laser lines can be extracted from each part to achieve multi-line laser separation.

[0040] The intrinsic parameter matrix and the calibration laser curve mapping relationship need to be calibrated. Specifically, image sets of the chessboard grid in different poses are obtained. The image sets include a first calibration image and a second calibration image. The first calibration image is an image of the chessboard grid without laser projection, and the second calibration image is an image of the chessboard grid with laser projection. Based on multiple image sets, the laser projection 3D reconstruction system is calibrated to obtain the camera's intrinsic parameter matrix and the calibration laser curve mapping relationship.

[0041] For example, to accurately reconstruct the contour point cloud of the region where the laser line is located, laser plane calibration and laser line center extraction are required beforehand. During laser plane calibration (i.e., the determination of the intrinsic parameter matrix and the mapping relationship of the calibration laser curve), theoretically, a single laser source projects a straight line outward, forming an absolute plane in space. The exemplary technology uses this assumption to calibrate and solve the equation of this plane. However, in reality, laser sources are affected by processing errors during manufacturing, and the projected laser line is not an absolute straight line, exhibiting a certain degree of curvature. Therefore, in this embodiment, the plane where the laser line is located is described using a spatial quadratic surface equation during calibration. To solve the quadratic surface equation, a checkerboard calibration board is used. After fixing a pose, images of the calibration board are taken with and without laser projection. Then, the pose of the calibration board is changed to obtain multiple sets of images. The camera intrinsic parameter matrix K and the laser surface equation are calculated using the obtained multiple sets of images. ,in The numbers represent the sequence of the four laser lines, and the equation of the laser surface is the calibrated mapping relationship of the laser curve.

[0042] After obtaining the calibration results, for the actual acquired scene images containing laser lines, firstly, the pixel regions (image matrix) where each laser line is located are separated; secondly, the center pixel coordinates of each laser line are extracted using the gray-scale centroid method; then, the corresponding laser surface equations are combined with the calibration results. In addition to the camera intrinsic parameter matrix K, four contour point clouds are obtained, meaning each image matrix contains two contour point clouds, and the laser projection image contains four contour point clouds. It should be noted that the contour point cloud refers to the image contour formed by the various boundary points in the laser projection image. For example, the laser projection image has four edges, and the four edges form the contour of the laser projection. Since the edges are composed of points, each edge is a contour point cloud.

[0043] Step S203: The image matrix containing the contour point cloud is filled in using the visual basic model to obtain the corresponding depth map of the laser projection image, and the depth map is converted into an image containing a global dense point cloud.

[0044] After obtaining the four contour point clouds of the laser projection image, in order to achieve overall 3D perception of the current field of view, it is necessary to complete the global dense point cloud of other regions in the laser projection image. To do this, the image matrix containing the contour point clouds is filled in using the visual baseline model set in the system, thus obtaining the depth map of the laser projection image. This depth map is then converted into an image of the global dense point cloud. The depth value of each pixel in the depth map represents the distance from that pixel to the camera.

[0045] For example, a visual base model can be obtained by training a deep learning model using training data. The training data includes contour images and images containing dense point clouds. An image with dense point clouds means that it contains not only points that form contours, but also many points that are not contours within those contours. The deep learning model performs point cloud completion on the contour images in the training data to obtain a completed image. This completed image is then compared with the images in the training data containing dense point clouds to guide the training of the deep learning model, thereby obtaining the visual base model.

[0046] For example, the contour point cloud is adjusted to the same resolution as the camera to obtain an intermediate image; the intermediate image is then input into a visual base model, which is used to perform point cloud completion on the intermediate image to obtain the corresponding depth map of the laser projection image.

[0047] Furthermore, the visual base model performs feature extraction on the intermediate image at multiple stages, and constructs the target image features based on the image features extracted at each stage; the image features extracted by the visual base model in the final stage of the intermediate image are added to the target image features to obtain the depth map. The following explains the completion of the contour point cloud. After the camera acquires the image, the system outputs a 2D image containing laser lines. In addition, it includes point clouds of multi-laser line contours. The system employs a multi-stage visual baseline model to extract image features from 2D images. The decoder in the system uses recombination operations to reassemble image features from different stages into an image-like representation. The visual base model can be a VIT (Vision Transformer) model or a SAM (Segment Anything Model); the decoder can be a DPT (Dense Prediction Transformer). Simultaneously, multiple contour point clouds are adjusted to a depth map with the same resolution as the 2D image. Then, this adjusted depth map is input into a shallow convolutional network to extract depth features. The extracted depth features are then projected onto the base image features through a zero-initialized convolutional layer. Same dimensions. Finally, deep features were added. In the process, the depth map is used for output, and then the depth map is converted into a point cloud to obtain a global dense point cloud, which is to convert the depth value of the pixel in the depth map into the expression of the point.

[0048] Reference Figure 3 ,Will Figure 3 (a) contains a laser projection image of the laser line and the contour point cloud, which is input into the visual base model. The visual base model completes the image using the point cloud to obtain... Figure 3 (b) shows the global depth map, which can be further transformed to obtain... Figure 3 Image (c) contains a globally dense point cloud. A globally dense point cloud image refers to a set of three-dimensional point clouds in a unified world coordinate system, consisting of points detected by radar, covering the entire scene, with high density, uniform distribution, and no voids.

[0049] In this embodiment, the camera is located at the center of a quadrilateral formed by laser light sources. The laser light sources positioned horizontally emit laser light of the same wavelength, while those positioned vertically emit laser light of different wavelengths. The system acquires the laser projection image captured by the camera and performs laser separation on the laser projection image to obtain two image matrices. By using the camera's intrinsic parameter matrix and the calibrated laser curve mapping relationship, each image matrix is ​​processed to obtain the contour point cloud of the laser light emitted by the laser light source in the image matrix. Then, the image matrix containing the contour point cloud is filled in using a visual basic model to obtain the depth map corresponding to the laser projection image. Finally, the depth map is converted into an image containing a globally dense point cloud, thereby obtaining a reconstructed image with high frame rate and high precision of the global field of view.

[0050] This application also provides a laser projection 3D reconstruction system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above-described method. This laser projection 3D reconstruction system can implement various embodiments of the above-described global depth estimation method based on multi-line lasers and achieve the same beneficial effects; further details are omitted here.

[0051] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the multi-line laser global depth estimation method according to various embodiments of this application as described in any of the foregoing embodiments of this specification.

[0052] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the power device, as a standalone firmware package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0053] Furthermore, embodiments of this application may also be computer-readable storage media storing a computer program thereon, the computer program being executed by a processor to perform the steps in the multi-line laser global depth estimation method according to various embodiments of this application described in any of the above embodiments of this specification, specifically implementing the steps of the multi-line laser global depth estimation method as described above.

[0054] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A global depth estimation method based on multi-line laser, characterized in that, An application is made in a laser projection 3D reconstruction system, which includes a camera and four laser light sources. The camera is located at the center of a quadrilateral formed by the laser light sources. The horizontally positioned laser light sources emit laser light of the same wavelength, while the vertically positioned laser light sources emit laser light of different wavelengths. The global depth estimation method based on multi-line lasers includes: Acquire the laser projection image captured by the camera, the laser projection image being formed by lasers emitted from each of the laser light sources; The laser projection image is separated into two image matrices. Based on the camera's intrinsic parameter matrix and the calibration laser curve mapping relationship, each image matrix is ​​processed to obtain the contour point cloud of the laser emitted by the laser source in each image matrix. The image matrix containing the contour point cloud is filled with point cloud by using a visual basic model to obtain the corresponding depth map of the laser projection image, and the depth map is converted into an image containing a globally dense point cloud.

2. The global depth estimation method based on multi-line laser as described in claim 1, characterized in that, Based on the camera's intrinsic parameter matrix and the calibrated laser curve mapping relationship, each image matrix is ​​processed to obtain the contour point cloud of the laser emitted by the laser source in each image matrix, including: Each image matrix is ​​divided into four equal parts to obtain multiple sub-images corresponding to the image matrix; Based on the camera's intrinsic parameter matrix and the calibration laser curve mapping relationship, each sub-image is processed to obtain four contour point clouds corresponding to each image matrix.

3. The global depth estimation method based on multi-line laser as described in claim 2, characterized in that, The step of dividing each of the image matrices into four equal parts includes: The image matrix is ​​divided equally in the horizontal direction and equally in the vertical direction.

4. The global depth estimation method based on multi-line laser as described in claim 1, characterized in that, Before acquiring the laser projection image captured by the camera, the method further includes: A set of images of a chessboard in different poses is obtained. The set of images includes a first calibration image and a second calibration image. The first calibration image is an image of the chessboard without laser projection, and the second calibration image is an image of the chessboard including laser projection. Based on the multiple sets of images, the laser projection 3D reconstruction system is calibrated to obtain the camera's intrinsic parameter matrix and the calibration laser curve mapping relationship.

5. The global depth estimation method based on multi-line laser as described in claim 1, characterized in that, The step of completing the image matrix containing the contour point cloud using a visual base model includes: The contour point cloud is adjusted to the same resolution as the camera to obtain an intermediate image; The intermediate image is input into the visual base model, and the intermediate image is filled in with point cloud data through the visual base model to obtain the corresponding depth map of the laser projection image.

6. The global depth estimation method based on multi-line laser as described in claim 5, characterized in that, The visual base model performs feature extraction on the intermediate image at multiple stages, and constructs target image features based on the image features extracted at each stage; The image features extracted from the final stage of the intermediate image by the visual base model are added to the target image features to obtain the depth map.

7. The global depth estimation method based on multi-line laser according to any one of claims 1-6, characterized in that, The laser source positioned horizontally emits a first laser, and the laser source positioned vertically emits a second laser, wherein the first laser and the second laser are different; or, the laser source positioned vertically emits a first laser, and the laser source positioned horizontally emits a second laser.

8. A laser projection three-dimensional reconstruction system, characterized in that, The system includes a camera and four laser light sources. The camera is located at the center of a quadrilateral formed by the laser light sources. The laser light sources arranged horizontally emit laser light of the same wavelength, while the laser light sources arranged vertically emit laser light of different wavelengths. The laser projection 3D reconstruction system also includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of any of the methods described in claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-7.

Images