Three-dimensional panoramic stitching method and system based on binocular vision, electronic equipment and storage medium

Abstract

The invention discloses a stereoscopic panoramic stitching method and system based on binocular vision, electronic equipment and a storage medium, the method is applied to the stereoscopic panoramic stitching system based on binocular vision, the system comprises a rotating platform and a binocular camera, and the binocular camera is arranged at the center of the rotating platform and rotates along with rotation of the rotating platform; the method comprises the following steps: in the rotating process of a rotating platform, continuously acquiring N binocular image pairs of a measured scene based on a binocular camera; converting the N binocular image pairs into N three-dimensional point cloud data sets; the N pieces of three-dimensional point cloud data comprise a target point cloud data set and N-1 to-be-registered point cloud data sets; and carrying out iteration registration splicing on the target point cloud data set and the N-1 to-be-registered point cloud data sets based on an acquisition sequence of the N binocular image pairs to obtain a panoramic point cloud data set of the detected scene. According to the invention, registration and splicing of the panoramic point cloud data can be realized, and the precision of the panoramic point cloud data is improved while the cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a binocular vision-based stereoscopic panorama stitching method, system, electronic device and storage medium. Background Art

[0002] Point cloud is the raw output data of most 3D information acquisition devices. Compared with 2D images, point cloud data can provide a more comprehensive description of real-world scenes. In recent years, with the improvement of computer hardware performance and the popularization of point cloud data acquisition devices, point cloud data utilization technology has been increasingly used in the fields of intelligent robots, unmanned driving, advanced manufacturing, virtual reality and augmented reality.

[0003] At present, the main method to obtain a complete 3D point cloud of the measured scene is to use LiDAR, depth camera or structured light camera to obtain multiple point clouds for point cloud fusion and stitching. However, the cost of obtaining point cloud data through LiDAR, depth camera or structured light camera is high, and the accuracy and robustness of the above methods for stitching the complete 3D point cloud of the measured scene are poor. Summary of the invention

[0004] The present invention provides a binocular vision-based stereoscopic panorama stitching method, system, electronic device and storage medium to solve the above technical problems.

[0005] A stereoscopic panoramic stitching method based on binocular vision according to the present invention is applied to a stereoscopic panoramic stitching system based on binocular vision, wherein the stereoscopic panoramic stitching system based on binocular vision comprises a rotating platform and a binocular camera, wherein the binocular camera is arranged at the center of the rotating platform and rotates with the rotation of the rotating platform; the method comprises:

[0006] During the rotation of the rotating platform, continuously acquiring N pairs of binocular images of the measured scene based on the binocular camera;

[0007] Convert the N binocular image pairs into N three-dimensional point cloud data sets; the N three-dimensional point cloud data sets include a target point cloud data set and N-1 point cloud data sets to be registered;

[0008] The target point cloud dataset and the N-1 point cloud datasets to be registered are iteratively registered and spliced ​​based on the acquisition order of the N binocular image pairs to obtain a panoramic point cloud dataset of the measured scene.

[0009] According to another aspect of the present invention, a stereoscopic panoramic stitching system based on binocular vision is provided, comprising: a rotating platform, a binocular camera and a point cloud stitching module, wherein the binocular camera is arranged at the center of the rotating platform and rotates with the rotation of the rotating platform, and the binocular camera is connected to the point cloud stitching module;

[0010] The binocular camera is used to continuously acquire N binocular image pairs of the measured scene during the rotation of the rotating platform, and transmit the N binocular image pairs to the point cloud stitching module;

[0011] The point cloud stitching module is used to receive the N binocular image pairs and convert the N binocular image pairs into N three-dimensional point cloud data sets; the N three-dimensional point cloud data include a target point cloud data set and N-1 point cloud data sets to be registered. The point cloud stitching module is used to iteratively register and stitch the target point cloud data set and the N-1 point cloud data sets to be registered based on the acquisition order of the N binocular image pairs to obtain a panoramic point cloud data set of the measured scene.

[0012] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the stereoscopic panoramic stitching method based on binocular vision described in any embodiment of the present invention.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the stereoscopic panoramic stitching method based on binocular vision described in any embodiment of the present invention when executed.

[0017] The technical solution of the embodiment of the present invention is to continuously acquire N binocular image pairs of the measured scene based on the binocular camera set at the center of the rotating platform during the rotation of the rotating platform; convert the N binocular image pairs into N three-dimensional point cloud data sets; the N three-dimensional point cloud data sets include a target point cloud data set and N-1 point cloud data sets to be registered; and iteratively register and splice the target point cloud data set and the N-1 point cloud data sets to be registered based on the acquisition order of the N binocular image pairs to obtain a panoramic point cloud data set of the measured scene. The registration and splicing of the panoramic point cloud data can be realized, and the accuracy of the panoramic point cloud data can be improved while reducing the cost.

[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a flow chart of a stereoscopic panoramic stitching method based on binocular vision provided in the first embodiment of the present invention;

[0021] Figure 2 is a position relationship diagram of a rotating platform and a binocular camera in a stereoscopic panoramic stitching system based on binocular vision provided in the first embodiment of the present invention;

[0022] Figure 3 is a schematic diagram of the coordinate system of the point cloud data set provided in the first embodiment of the present invention changing as the rotating platform rotates;

[0023] Figure 4 1 is a schematic diagram of the structure of a stereoscopic panoramic stitching system based on binocular vision provided in Embodiment 2 of the present invention;

[0024] Figure 5 It is a structural schematic diagram of an electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] Embodiment 1

[0028] Figure 1 It is a flow chart of a stereoscopic panoramic stitching method based on binocular vision provided in Example 1 of the present invention. This embodiment is applicable to the situation where the three-dimensional point cloud data of the measured scene is registered and stitched to obtain the panoramic point cloud data of the measured scene. The method can be executed by a stereoscopic panoramic stitching system based on binocular vision. The stereoscopic panoramic stitching system based on binocular vision can be implemented in the form of hardware and / or software. The stereoscopic panoramic stitching system based on binocular vision includes a rotating platform and a binocular camera.

[0029] Figure 2 : is a position relationship diagram of the rotating platform and the binocular camera in the stereo panoramic stitching system based on binocular vision provided in the first embodiment of the present invention, such as Figure 2 As shown, the binocular camera is arranged at the center of the rotating platform and rotates with the rotating platform. It can be understood that in order to ensure that the binocular camera can work stably and reliably, the midpoint of the line connecting the two optical centers of the binocular camera is located at the center point of the rotating platform.

[0030] like Figure 1 As shown, the method includes:

[0031] S110. During the rotation of the rotating platform, continuously acquire N pairs of binocular images of the measured scene based on the binocular camera.

[0032] Among them, a binocular image pair refers to an image pair formed by two images taken by two cameras of a binocular camera at the same time. In this embodiment, during the rotation of the rotating platform, the binocular camera continuously captures images of the measured scene and forms N binocular image pairs. Among them, the shooting interval is set by those skilled in the art according to specific scenes and requirements, and is not limited here. It can be understood that the images in two adjacent binocular image pairs partially overlap.

[0033] It should be noted that, before the rotating platform rotates, the parameters of the rotating platform can be set according to the actual measured scene, and the data acquisition scheme can be flexibly adjusted to meet different application requirements.

[0034] In some embodiments, optionally, after continuously acquiring N binocular image pairs based on the binocular camera, the method further includes: preprocessing the N binocular image pairs respectively based on pre-calibrated camera parameters of the binocular camera; the preprocessing includes one or more of dedistortion and correction.

[0035] The camera parameters include but are not limited to camera intrinsic parameters, distortion coefficients, etc. In this embodiment, based on pre-calibrated camera intrinsic parameters, distortion coefficients and other parameters, N binocular image pairs are respectively dedistorted and corrected to ensure the accuracy of the three-dimensional point cloud data obtained by subsequent conversion.

[0036] S120, converting the N binocular image pairs into N three-dimensional point cloud data sets; the N three-dimensional point cloud data sets include a target point cloud data set and N-1 point cloud data sets to be registered.

[0037] In this embodiment, N binocular image pairs acquired by the binocular camera are converted into N three-dimensional point cloud data sets. Among them, the N three-dimensional point cloud data sets include a target point cloud data set and N-1 point cloud data sets to be registered; specifically, the target point cloud data set is used as the reference for registration, and is the three-dimensional point cloud data set converted from the binocular image pair taken for the first time. Therefore, the target point cloud data set can be named C0; correspondingly, the N-1 point cloud data sets to be registered can be named according to the shooting order, and can be named in sequence: C1, C2, ..., C N-1 .

[0038] Based on the above embodiment, optionally, converting the N binocular image pairs into N three-dimensional point cloud data sets includes: performing stereo matching on the N binocular image pairs respectively to obtain N depth images; for any image in each of the binocular image pairs, converting the image into a three-dimensional point cloud data set based on the camera intrinsic parameters of the binocular camera and the depth value of the depth image.

[0039] In this embodiment, the SGBM (Semi-Global Block Matching) algorithm is used to perform stereo matching on N binocular image pairs to generate N depth images, where each depth image corresponds to a shooting position on the rotating platform. For any image in each binocular image pair, the pixel points of the image are expanded into three-dimensional point cloud data using the camera intrinsic parameters of the binocular camera and the depth value of the depth image, thereby obtaining N three-dimensional point cloud data sets. Among them, the camera intrinsic parameters include but are not limited to parameters such as focal length and principal point position. It should be noted that any image in the binocular image pair can be a left image or a right image, but the left image is usually selected to avoid occlusion problems.

[0040] Exemplarily, the three-dimensional coordinate transformation formula is:

[0041] , ,Z=depth value of the depth image.

[0042] Among them, (u, v) is the pixel coordinate on the image, (X, Y, Z) is the corresponding three-dimensional space coordinate, and f x and f y is the focal length, (c x ,c y ) as the main point position.

[0043] In some embodiments, after obtaining the three-dimensional point cloud data, the point cloud data in each three-dimensional point cloud data set is optionally downsampled and denoised. Downsampling can reduce the number of points in the point cloud, thereby reducing the amount of calculation for subsequent processing; denoising can remove abnormal points caused by noise, mismatching, etc.

[0044] S130, iteratively registering and stitching the target point cloud dataset and the N-1 point cloud datasets to be registered based on the acquisition order of the N binocular image pairs, to obtain a panoramic point cloud dataset of the measured scene.

[0045] In this embodiment, the target point cloud data and the first point cloud data set to be registered are registered and spliced ​​to obtain the first spliced ​​point cloud data set; further, the first spliced ​​point cloud data set and the second point cloud data set to be registered are registered and spliced ​​to obtain the second spliced ​​point cloud data set, and the registration and splicing are iteratively performed according to the acquisition order of N binocular image pairs until the N-1th spliced ​​point cloud data set is obtained, and the N-1th spliced ​​point cloud data set is determined as the panoramic point cloud data set of the measured scene.

[0046] On the basis of the above embodiment, optionally, the target point cloud dataset and the N-1 point cloud datasets to be registered are iteratively registered and spliced ​​according to the acquisition order of the N binocular image pairs to obtain a panoramic point cloud dataset of the measured scene, including: coarsely registering the i-th point cloud dataset to be registered and the i-1-th spliced ​​point cloud dataset to obtain an initial transformation matrix from the i-th point cloud dataset to be registered to the i-1-th spliced ​​point cloud dataset; wherein the spliced ​​point cloud dataset of the first iteration is the target point cloud dataset; taking the initial transformation matrix as the starting point, the i-th point cloud dataset to be registered and the i-1-th spliced ​​point cloud dataset are coarsely registered. Perform precise registration to obtain the target transformation matrix of the i-th point cloud dataset to be registered and the i-1-th stitched point cloud dataset; based on the target transformation matrix, convert the i-th point cloud dataset to be registered to the coordinate system of the i-1-th stitched point cloud dataset to obtain the i-th point cloud dataset to be stitched; stitch the i-th point cloud dataset to be stitched and the i-1-th stitched point cloud dataset to obtain the i-th stitched point cloud dataset; iteratively perform the above steps based on the acquisition order of the N binocular image pairs, and when the number of executions is N-1 times, determine the N-1-th stitched point cloud dataset as the panoramic point cloud dataset of the measured scene.

[0047] In this embodiment, the process of iterative registration and stitching is as follows:

[0048] 1) Roughly align the i-th point cloud dataset to be registered and the i-1-th spliced ​​point cloud dataset to obtain the initial transformation matrix from the i-th point cloud dataset to be registered to the i-1-th spliced ​​point cloud dataset; where the spliced ​​point cloud dataset refers to the point cloud data obtained by aligning and splicing the point cloud dataset to be registered in the current iteration and the spliced ​​point cloud dataset in the previous iteration. It should be noted that the spliced ​​point cloud dataset of the first iteration is the target point cloud dataset.

[0049] 2) Taking the initial transformation matrix as the starting point, the i-th point cloud dataset to be registered and the i-1-th spliced ​​point cloud dataset are precisely registered to obtain the target transformation matrix of the i-th point cloud dataset to be registered and the i-1-th spliced ​​point cloud dataset.

[0050] 3) Based on the target transformation matrix, the i-th point cloud data set to be registered is transformed into the coordinate system of the i-1-th spliced ​​point cloud data set to obtain the i-th point cloud data set to be spliced; wherein, the i-th point cloud data set to be spliced ​​refers to the point cloud data obtained by transforming the i-th point cloud data set to be registered into the coordinate system of the spliced ​​point cloud data set. It should be noted that the point cloud data to be spliced, the spliced ​​point cloud data and the target point cloud data are in the same coordinate system.

[0051] 4) Splicing the i-th point cloud dataset to be spliced ​​and the i-1-th splicing point cloud dataset to obtain the i-th splicing point cloud dataset;

[0052] 5) Iterate the above steps 1)-4 based on the acquisition order of N binocular image pairs. When the number of executions is N-1, the N-1th spliced ​​point cloud dataset is determined as the panoramic point cloud dataset of the measured scene. It can be understood that when the number of executions is N-1, all N-1 point cloud datasets to be registered are registered and spliced ​​into the coordinate system where the target point cloud dataset is located.

[0053] On the basis of the above embodiment, optionally, for the i-th point cloud dataset to be registered, the i-th point cloud dataset to be registered and the i-1-th stitched point cloud dataset are roughly registered to obtain an initial transformation matrix from the i-th point cloud dataset to be registered to the i-1-th stitched point cloud dataset, including: obtaining the rotation angle of the rotating platform from the target point cloud dataset to the i-th point cloud dataset to be registered; determining the initial transformation matrix from the i-th point cloud dataset to be registered to the i-1-th stitched point cloud dataset based on the rotation angle and the optical center distance of the binocular camera; the stitched point cloud dataset and the target point cloud dataset are in the same coordinate system.

[0054] The rotation angle refers to the angle that the rotating platform rotates from the initial position to the current position. The initial position is the optical center of the left camera or the optical center of the camera when the binocular image corresponding to the target point cloud dataset is taken. The current position is the optical center of the left camera or the optical center of the camera when the binocular image corresponding to the i-th point cloud dataset to be registered is taken. Specifically, the rotation angle can be directly read by the encoder or sensor of the rotating platform, or it can be recorded each time the binocular image is taken. Each shot is recorded in turn, and the rotation angle of the rotating platform relative to the initial position during this shot is recorded. The optical center distance refers to the horizontal distance between the optical centers of the two camera lenses. The optical center distance is fixed, also known as the baseline distance.

[0055] In this embodiment, the rotation angle of the rotating platform from the target point cloud dataset to the i-th point cloud dataset to be registered is obtained. According to the rotation angle and the optical center distance of the binocular camera, the initial transformation matrix from the i-th point cloud dataset to be registered to the target point cloud dataset can be determined. It can be understood that since the spliced ​​point cloud dataset and the target point cloud dataset are in the same coordinate system, the initial transformation matrix from the i-th point cloud dataset to be registered to the target point cloud dataset is the initial transformation matrix from the i-th point cloud dataset to be registered to the i-1-th spliced ​​point cloud dataset. Among them, the initial transformation matrix includes two parts: the translation matrix and the rotation matrix.

[0056] For example, Figure 3 Schematic diagram of the coordinate system of the point cloud data set provided in the first embodiment of the present invention changing as the rotating platform rotates. Figure 3 As shown, the Z axis of the coordinate system points to the optical axis, the X axis points to the center of the rotating platform, and the Y axis points perpendicularly to the rotating platform. When the rotating platform rotates, the coordinate system rotates around the Y axis. C0 is the coordinate system at the initial position, C i is the coordinate system of the current position, b is half of the distance of the binocular camera’s optical center, is the rotation angle of the rotating platform from the initial position to the current position, that is, the rotation angle of the rotating platform from the target point cloud dataset to the i-th point cloud dataset to be registered.

[0057] According to the rotation angle and the optical center distance of the binocular camera, the translation matrix and rotation matrix of the i-th point cloud data set to be registered to the target point cloud data set can be calculated respectively.

[0058] The translation matrix is:

[0059]

[0060] The rotation matrix is:

[0061]

[0062] It can be obtained that the initial transformation matrix is:

[0063]

[0064] On the basis of the above embodiment, optionally, taking the initial transformation matrix as the starting point, accurately aligning the i-th point cloud dataset to be registered and the i-1-th spliced ​​point cloud dataset to obtain the target transformation matrix of the i-th point cloud dataset to be registered and the i-1-th spliced ​​point cloud dataset, including: transforming the i-th point cloud dataset to be registered based on the current transformation matrix to obtain an intermediate point cloud dataset; wherein the current transformation matrix at the first iteration is the initial transformation matrix; determining corresponding point pairs in the intermediate point cloud dataset and the i-1-th spliced ​​point cloud dataset based on a preset search method; determining the intermediate transformation matrix based on the corresponding point pairs using a preset optimization method; updating the current transformation matrix based on the intermediate transformation matrix; iteratively executing the above steps, and determining the current transformation matrix as the target transformation matrix when the convergence conditions are met.

[0065] In this embodiment, the initial transformation matrix iterative optimization process is as follows:

[0066] 1) Based on the current transformation matrix, transform the i-th point cloud dataset to be registered to obtain an intermediate point cloud dataset; if this iteration is the first iteration, the current transformation matrix is ​​the initial transformation matrix.

[0067] 2) Determine corresponding point pairs in the intermediate point cloud dataset and the i-1th spliced ​​point cloud dataset based on a preset search method; wherein the preset search method includes but is not limited to nearest neighbor search, point-to-point matching in ICP, normal alignment matching, etc., which are not limited here.

[0068] 3) Based on the corresponding point pairs, the intermediate transformation matrix is ​​determined using a preset optimization method; the preset optimization method includes but is not limited to ICP, SVD solution, nonlinear optimization, etc., which are not limited here. The intermediate transformation matrix is ​​the transformation matrix optimized by the preset optimization method.

[0069] 4) Update the current transformation matrix based on the intermediate transformation matrix; specifically, the current matrix can be multiplied by the intermediate transformation matrix to obtain a new transformation matrix, and the new transformation matrix is ​​used as the current change matrix for the next iteration.

[0070] 4) Iterate the above steps, and determine the current transformation matrix as the target transformation matrix when the convergence condition is met. The convergence condition includes but is not limited to the number of iterations reaching a preset iteration threshold, the average distance between corresponding point pairs being less than a preset distance threshold, and the change in the transformation matrix being less than a second change threshold.

[0071] On the basis of the above embodiment, optionally, after obtaining the i-th spliced ​​point cloud data set, the method further includes: traversing the i-th spliced ​​point cloud data set to determine the distance between any two point cloud data; when the distance between two point cloud data is less than a preset overlapping point determination threshold, determining the two point cloud data as an overlapping point pair, and deleting any point cloud data in the overlapping point pair.

[0072] The preset overlapping point determination threshold is a distance threshold between two point cloud data in the spliced ​​point cloud data set, and is used to determine whether the point cloud data in the spliced ​​point cloud data set is an overlapping point. In this embodiment, the point cloud data in the i-th spliced ​​point cloud data set is traversed, and for any two point cloud data encountered during the traversal process, the Euclidean distance between the two point cloud data is calculated, and the calculated Euclidean distance is compared with the preset overlapping point determination threshold. If the distance is less than the preset overlapping point determination threshold, it is considered that the two point cloud data are overlapping, and the two point cloud data are determined as an overlapping point pair, and any point cloud data in the overlapping point pair is deleted to avoid redundancy of overlapping point cloud data.

[0073] Exemplarily, the overlapping point determination formula is as follows:

[0074]

[0075] in, represents the preset overlapping point determination threshold, (x1, y1, z1) and (x2, y2, z2) represent the two traversed point cloud data respectively.

[0076] The technical solution of this embodiment is to continuously acquire N binocular image pairs of the measured scene based on the binocular camera set at the center of the rotating platform during the rotation of the rotating platform; convert the N binocular image pairs into N three-dimensional point cloud data sets; the N three-dimensional point cloud data sets include the target point cloud data set and N-1 point cloud data sets to be registered; and iteratively register and splice the target point cloud data set and the N-1 point cloud data sets to be registered based on the acquisition order of the N binocular image pairs to obtain the panoramic point cloud data set of the measured scene. The registration and splicing of the panoramic point cloud data can be realized, and the accuracy of the panoramic point cloud data can be improved while reducing the cost.

[0077] Embodiment 2

[0078] Figure 4 Schematic diagram of the structure of a stereoscopic panoramic stitching system based on binocular vision provided by the second embodiment of the present invention. Figure 4 As shown, the system includes: a rotating platform 310, a binocular camera 320 and a point cloud stitching module 330, wherein the binocular camera is arranged at the center of the rotating platform and rotates with the rotation of the rotating platform 310, and the binocular camera 320 and the point cloud stitching module 330 are connected;

[0079] The binocular camera 320 is used to continuously acquire N binocular image pairs of the measured scene during the rotation of the rotating platform 310, and transmit the N binocular image pairs to the point cloud stitching module 330;

[0080] The point cloud stitching module 330 is used to receive the N binocular image pairs and convert the N binocular image pairs into N three-dimensional point cloud data sets; the N three-dimensional point cloud data include a target point cloud data set and N-1 point cloud data sets to be registered. The point cloud stitching module repeatedly performs registration and stitching on the target point cloud data set and the N-1 point cloud data sets to be registered based on the acquisition order of the N binocular image pairs to obtain a panoramic point cloud data set of the measured scene.

[0081] It should be noted that in actual applications, the point cloud stitching module 330 can be set in the processor of the binocular camera 320, or in an electronic device connected to the binocular camera 320. It is set by technical personnel in this field according to needs and is not specifically limited here.

[0082] The technical solution of this embodiment is to continuously acquire N binocular image pairs of the measured scene based on the binocular camera set at the center of the rotating platform during the rotation of the rotating platform; convert the N binocular image pairs into N three-dimensional point cloud data sets; the N three-dimensional point cloud data sets include the target point cloud data set and N-1 point cloud data sets to be registered; and iteratively register and splice the target point cloud data set and the N-1 point cloud data sets to be registered based on the acquisition order of the N binocular image pairs to obtain the panoramic point cloud data set of the measured scene. The registration and splicing of the panoramic point cloud data can be realized, and the accuracy of the panoramic point cloud data can be improved while reducing the cost.

[0083] Based on the above embodiment, optionally, the point cloud stitching module 330 includes a three-dimensional point cloud data set conversion unit, which is used to perform stereo matching on the N binocular image pairs respectively to obtain N depth images; for any image in each of the binocular image pairs, the image is converted into a three-dimensional point cloud data set based on the camera intrinsic parameters of the binocular camera and the depth value of the depth image.

[0084] On the basis of the above embodiment, optionally, the point cloud stitching module 330 includes a registration stitching unit, which is used to perform a rough registration on the i-th point cloud data set to be registered and the i-1th stitching point cloud data set for the i-th point cloud data set to be registered, and obtain an initial transformation matrix from the i-th point cloud data set to be registered to the i-1th stitching point cloud data set; wherein the i-1th stitching point cloud data set corresponding to the first point cloud data set to be registered is the target point cloud data set; taking the initial transformation matrix as the starting point, the i-th point cloud data set to be registered and the i-1th stitching point cloud data set are finely registered to obtain the The target transformation matrix of the i-th point cloud dataset to be registered and the i-1-th stitching point cloud dataset is obtained; based on the target transformation matrix, the i-th point cloud dataset to be registered is converted to the coordinate system of the i-1-th stitching point cloud dataset to obtain the i-th point cloud dataset to be stitched; the i-th point cloud dataset to be stitched and the i-1-th stitching point cloud dataset are stitched to obtain the i-th stitching point cloud dataset; the above steps are repeatedly performed based on the acquisition order of the N binocular image pairs, and when the number of executions is N-1 times, the N-1-th stitching point cloud dataset is determined as the panoramic point cloud dataset of the measured scene.

[0085] Based on the above embodiment, optionally, the registration and stitching unit includes a coarse registration subunit, which is used to obtain the rotation angle of the rotating platform from the target point cloud dataset to the i-th point cloud dataset to be registered; based on the rotation angle and the optical center distance of the binocular camera, determine the initial transformation matrix from the i-th point cloud dataset to be registered to the i-1-th stitching point cloud dataset; the stitching point cloud dataset and the target point cloud dataset are in the same coordinate system.

[0086] On the basis of the above embodiment, optionally, the registration and stitching unit includes a fine registration subunit, which is used to transform the i-th point cloud data set to be registered based on the current transformation matrix to obtain an intermediate point cloud data set; wherein the current transformation matrix at the first iteration is the initial transformation matrix; corresponding point pairs are determined in the intermediate point cloud data set and the i-1-th stitched point cloud data set based on a preset search method; based on the corresponding point pairs, an intermediate transformation matrix is ​​determined using a preset optimization method; the current transformation matrix is ​​updated based on the intermediate transformation matrix; the above steps are iteratively performed, and when the convergence conditions are met, the current transformation matrix is ​​determined as the target transformation matrix.

[0087] Based on the above embodiment, optionally, the point cloud registration module also includes an overlapping point deletion unit, which is used to traverse the i-th spliced ​​point cloud data set to determine the distance between any two point cloud data; when the distance between two point cloud data is less than a preset overlapping point determination threshold, the two point cloud data are determined as an overlapping point pair, and any point cloud data in the overlapping point pair is deleted.

[0088] Based on the above embodiment, optionally, the point cloud registration module also includes a preprocessing unit, which is used to preprocess the N binocular image pairs respectively based on the pre-calibrated camera parameters of the binocular camera; the preprocessing includes one or more of dedistortion and correction.

[0089] Embodiment 3

[0090] Figure 5 1 is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0091] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0092] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0093] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a stereoscopic panoramic stitching method based on binocular vision.

[0094] In some embodiments, the stereoscopic panoramic stitching method based on binocular vision can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the stereoscopic panoramic stitching method based on binocular vision described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute the stereoscopic panoramic stitching method based on binocular vision in any other appropriate manner (for example, by means of firmware).

[0095] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0096] The computer program for implementing the binocular vision-based stereoscopic panoramic stitching method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flowchart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine, or completely on a remote machine or server.

[0097] Embodiment 4

[0098] Embodiment 4 of the present invention further provides a computer-readable storage medium, the computer-readable storage medium stores computer instructions, the computer instructions are used to enable a processor to execute a stereoscopic panoramic stitching method based on binocular vision, the method is applied to a stereoscopic panoramic stitching system based on binocular vision, the stereoscopic panoramic stitching system based on binocular vision includes a rotating platform and a binocular camera, the binocular camera is arranged at the center of the rotating platform, and rotates with the rotation of the rotating platform; the method includes:

[0099] During the rotation of the rotating platform, N pairs of binocular images of the measured scene are continuously acquired based on the binocular camera;

[0100] Convert N binocular image pairs into N three-dimensional point cloud data sets; the N three-dimensional point cloud data sets include a target point cloud data set and N-1 point cloud data sets to be registered;

[0101] Based on the acquisition order of N binocular image pairs, the target point cloud dataset and N-1 point cloud datasets to be registered are repeatedly registered and spliced ​​to obtain the panoramic point cloud dataset of the measured scene.

[0102] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0103] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0104] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0105] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0106] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0107] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A stereoscopic panorama stitching method based on binocular vision, characterized in that: The invention is applied to a stereoscopic panoramic stitching system based on binocular vision, wherein the stereoscopic panoramic stitching system based on binocular vision comprises a rotating platform and a binocular camera, wherein the binocular camera is arranged at the center of the rotating platform and rotates with the rotation of the rotating platform; the method comprises: During the rotation of the rotating platform, continuously acquiring N pairs of binocular images of the measured scene based on the binocular camera; Convert the N binocular image pairs into N three-dimensional point cloud data sets; the N three-dimensional point cloud data sets include a target point cloud data set and N-1 point cloud data sets to be registered; The target point cloud dataset and the N-1 point cloud datasets to be registered are iteratively registered and spliced ​​based on the acquisition order of the N binocular image pairs to obtain a panoramic point cloud dataset of the measured scene.

2. The method according to claim 1, characterized in that The converting the N binocular image pairs into N three-dimensional point cloud data sets includes: Performing stereo matching on the N binocular image pairs respectively to obtain N depth images; For any image in each of the binocular image pairs, the image is converted into a three-dimensional point cloud data set based on the camera intrinsic parameters of the binocular camera and the depth value of the depth image.

3. The method according to claim 1, characterized in that The iterative registration and splicing of the target point cloud dataset and the N-1 point cloud datasets to be registered according to the acquisition order of the N binocular image pairs to obtain a panoramic point cloud dataset of the measured scene includes: Roughly align the i-th point cloud dataset to be registered and the i-1-th spliced ​​point cloud dataset to obtain an initial transformation matrix from the i-th point cloud dataset to be registered to the i-1-th spliced ​​point cloud dataset; wherein the spliced ​​point cloud dataset of the first iteration is the target point cloud dataset; Taking the initial transformation matrix as a starting point, the i-th point cloud data set to be registered and the i-1-th spliced ​​point cloud data set are precisely registered to obtain a target transformation matrix of the i-th point cloud data set to be registered and the i-1-th spliced ​​point cloud data set; Based on the target transformation matrix, the i-th point cloud data set to be registered is converted into the coordinate system of the i-1-th spliced ​​point cloud data set to obtain the i-th point cloud data set to be spliced; Splicing the i-th point cloud data set to be spliced ​​and the i-1-th spliced ​​point cloud data set to obtain an i-th spliced ​​point cloud data set; The above steps are iteratively performed based on the acquisition order of the N binocular image pairs, and when the number of executions is N-1 times, the N-1th spliced ​​point cloud data set is determined as the panoramic point cloud data set of the measured scene.

4. The method according to claim 3, characterized in that: For the i-th point cloud data set to be registered, the i-th point cloud data set to be registered and the i-1-th spliced ​​point cloud data set are roughly registered to obtain an initial transformation matrix from the i-th point cloud data set to be registered to the i-1-th spliced ​​point cloud data set, including: Obtaining a rotation angle of the rotating platform from the target point cloud dataset to the i-th point cloud dataset to be registered; An initial transformation matrix from the i-th point cloud data set to be registered to the i-1-th spliced ​​point cloud data set is determined based on the rotation angle and the optical center distance of the binocular camera; the spliced ​​point cloud data set and the target point cloud data set are in the same coordinate system.

5. The method according to claim 3, characterized in that: The method of taking the initial transformation matrix as a starting point, performing precise registration on the i-th point cloud data set to be registered and the i-1-th spliced ​​point cloud data set, and obtaining a target transformation matrix of the i-th point cloud data set to be registered and the i-1-th spliced ​​point cloud data set, comprises: Transforming the i-th point cloud data set to be registered based on the current transformation matrix to obtain an intermediate point cloud data set; wherein the current transformation matrix at the first iteration is the initial transformation matrix; Determining corresponding point pairs in the intermediate point cloud dataset and the (i-1)th spliced ​​point cloud dataset based on a preset search method; Based on the corresponding point pairs, determining an intermediate transformation matrix using a preset optimization method; updating the current transformation matrix based on the intermediate transformation matrix; The above steps are iteratively performed, and when the convergence condition is met, the current transformation matrix is ​​determined as the target transformation matrix.

6. The method according to claim 3, characterized in that After obtaining the i-th spliced ​​point cloud data set, the method further includes: Traversing the i-th spliced ​​point cloud data set to determine the distance between any two point cloud data; When the distance between two point cloud data is less than a preset overlapping point determination threshold, the two point cloud data are determined as an overlapping point pair, and any point cloud data in the overlapping point pair is deleted.

7. The method according to claim 1, characterized in that After continuously acquiring N binocular image pairs based on the binocular camera, the method further includes: The N binocular image pairs are preprocessed respectively based on the pre-calibrated camera parameters of the binocular camera; the preprocessing includes one or more of dedistortion and correction.

8. A stereoscopic panoramic stitching system based on binocular vision, characterized in that: include: A rotating platform, a binocular camera and a point cloud stitching module, wherein the binocular camera is arranged at the center of the rotating platform and rotates with the rotation of the rotating platform, and the binocular camera is connected to the point cloud stitching module; The binocular camera is used to continuously acquire N binocular image pairs of the measured scene during the rotation of the rotating platform, and transmit the N binocular image pairs to the point cloud stitching module; The point cloud stitching module is used to receive the N binocular image pairs and convert the N binocular image pairs into N three-dimensional point cloud data sets; the N three-dimensional point cloud data include a target point cloud data set and N-1 point cloud data sets to be registered. The point cloud stitching module is used to iteratively register and stitch the target point cloud data set and the N-1 point cloud data sets to be registered based on the acquisition order of the N binocular image pairs to obtain a panoramic point cloud data set of the measured scene.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the stereoscopic panoramic stitching method based on binocular vision described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the stereoscopic panoramic stitching method based on binocular vision according to any one of claims 1 to 7 when executed.

Images