Tunnel three-dimensional holographic stitching method and device based on panoramic array camera network

Abstract

The present invention relates to a tunnel 3D holographic stitching method and device using a panoramic array camera network. The method uses a reference transfer camera array and a 3D laser scanner on an inspection vehicle to move with the inspection vehicle to different inspection positions for adjustment, thereby transmitting peripheral control information from both ends of the tunnel to the measurement area and constructing a dynamic control field, reducing restrictions on measurement conditions. High-precision positioning and attitude determination are performed based on the fixed constraints of the surround-view camera array and the 3D laser scanner and the constructed dynamic control field, thereby unifying the measurement platform into a global coordinate system, effectively avoiding the inherent defect of large cumulative errors in traditional 3D modeling methods. The high-resolution surround-view image is then mapped to 3D point cloud data in the global coordinate system, thereby realizing high-resolution and high-precision 3D holographic stitching of the tunnel annulus in the global coordinate system, while achieving global and local high resolution and high precision, greatly improving the acquisition accuracy of the tunnel 3D hologram.

Description

Technical Field

[0001] The present invention belongs to the technical field of large-scale civil structure health detection, and relates to a tunnel three-dimensional holographic splicing method and device using a panoramic array camera network. Background Art

[0002] Through high-precision data acquisition and complex image processing algorithms, a comprehensive, all-around view of the tunnel's internal environment is achieved, providing intuitive and comprehensive information support for tunnel design, construction, monitoring, and operation and maintenance. This not only improves the visualization level of tunnel projects, but also brings significant changes to tunnel safety monitoring, construction management, and education and training. Existing panoramic stitching technologies can be divided into 2D panoramic stitching and 3D panoramic stitching, depending on the presentation format of the output results. 2D panoramic stitching systems (such as CN202410665168.8 and CN202111455918.1) project images captured by the camera onto a top-down plane and stitch them together to generate a panoramic image. However, based on the assumption that the scene is coplanar, this system can only provide a single top-down perspective with a limited field of view. If there are vertical target objects distributed in the scene, significant stitching errors will occur. 3D panoramic stitching technology (such as CN202410261618.7) maps the stitched panorama onto a 3D model, allowing users to freely switch perspectives and viewpoints for panoramic roaming as needed. However, existing 3D stitching mostly maps the panorama to a 3D model with preset values ​​such as a column, bowl, or sphere. If the model deviates from the actual model, it will cause obvious stitching errors.

[0003] Existing panoramic stitching technologies are primarily designed for small passenger vehicles, focusing on utilizing multiple cameras distributed across different vehicle body locations to simultaneously capture panoramic images at each moment. However, for tunnel scenarios, local high-resolution panoramic images captured by panoramic cameras installed on inspection vehicles at different inspection locations must be stitched together to form a global, high-resolution surround view of the entire inspection area. Furthermore, in addition to segments, tunnel surfaces also contain structures such as pipes and lighting fixtures. These structures are typically at the decimeter level, making them significant in tunnels (particularly subway tunnels). Therefore, traditional cylindrical projection-based 3D panoramic stitching models are unsuitable. Furthermore, stitching local high-resolution panoramic images together to form a global high-resolution panoramic image can result in significant cumulative errors when control information is incomplete. Therefore, it is necessary to address the challenge of 3D high-resolution panoramic stitching of tunnel annular surfaces with limited control information. Summary of the Invention

[0004] In response to the problems existing in the above-mentioned traditional methods, the present invention proposes a tunnel three-dimensional holographic stitching method and a tunnel three-dimensional holographic stitching device using a panoramic array camera network, which can achieve high-resolution and high-precision three-dimensional holographic stitching of the tunnel annulus in a global coordinate system.

[0005] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0006] On the one hand, a method for stitching a three-dimensional hologram of a tunnel using a panoramic array camera network is provided, comprising the steps of:

[0007] Offline calibration of the panoramic camera array and 3D laser scanner on the inspection vehicle; the panoramic camera array includes a surround-view camera array and a reference transfer camera array. The reference transfer camera array includes a fixed-mounted forward-view camera and a rear-view camera, which are used to transmit control information from both ends of the tunnel to the measurement area. The surround-view camera array includes at least two high-resolution cameras with overlapping fields of view.

[0008] The inspection vehicle moves the panoramic camera array and 3D laser scanner to each inspection location. The surround-view camera array acquires high-resolution surround-view images of the tunnel at each inspection location. The fiducial transfer camera array takes photos and extracts the center coordinates of the measurement points within the measurement area. The 3D laser scanner collects dense 3D point cloud data within the measurement area.

[0009] Based on the center coordinates of the measurement points in the survey area and the control information at both ends of the tunnel, the camera array and the 3D laser scanner are transferred to perform joint adjustment of different inspection positions to construct a dynamic control field with known displacement.

[0010] Based on the dynamic control field, the position of the measurement platform relative to the global coordinates is estimated using the surround-view camera array and three or more dynamic control points observed by the 3D laser scanner. The 3D point cloud obtained by the 3D laser scanner is then stitched into the global coordinate system.

[0011] According to the fixed connection constraint between the surround-view camera array and the 3D laser scanner, the high-resolution surround-view image is mapped to 3D point cloud data in the global coordinate system to obtain a global high-resolution 3D holographic image of the tunnel.

[0012] On the other hand, a tunnel three-dimensional holographic stitching device using a panoramic array camera network is also provided, comprising:

[0013] An offline calibration module is used to perform offline calibration on the panoramic camera array and 3D laser scanner on the inspection vehicle. The panoramic camera array includes a surround-view camera array and a reference transfer camera array. The reference transfer camera array includes a rigidly mounted forward-view camera and a rear-view camera, which are used to transmit control information from both ends of the tunnel to the measurement area. The surround-view camera array includes at least two high-resolution cameras with overlapping fields of view.

[0014] The inspection acquisition module is used to move the panoramic camera array and 3D laser scanner to each inspection location via the inspection vehicle. The surround-view camera array is used to obtain high-resolution surround-view images of the tunnel at each inspection location. The reference transfer camera array is used to take photos and extract the center coordinates of the measurement points in the measurement area. The 3D laser scanner is used to collect dense 3D point cloud data within the measurement area.

[0015] The control field construction module is used to transfer the camera array and 3D laser scanner to perform joint adjustment of different inspection position benchmarks based on the center coordinates of the measurement points in the measurement area and the control information at both ends of the tunnel, thereby constructing a dynamic control field with known displacement;

[0016] The positioning and attitude determination module is used to estimate the position of the measurement platform relative to the global coordinate system based on the dynamic control field and three or more dynamic control points observed by the surround-view camera array and the 3D laser scanner, and then stitch the 3D point cloud scanned by the 3D laser scanner into the global coordinate system;

[0017] The holographic stitching module is used to map the high-resolution surround view image to three-dimensional point cloud data in the global coordinate system based on the fixed connection constraints between the surround view camera array and the three-dimensional laser scanner, so as to obtain a global high-resolution three-dimensional holographic image of the tunnel.

[0018] One of the above technical solutions has the following advantages and beneficial effects:

[0019] The above-mentioned panoramic array camera network-based tunnel 3D holographic stitching method and device utilizes a panoramic camera array and 3D laser scanner installed on an inspection vehicle. The reference transfer camera array and 3D laser scanner within the device move with the inspection vehicle to different inspection locations for adjustment. This method achieves high-precision transmission of peripheral control information from both ends of the tunnel to the measurement area, thereby constructing a dynamic control field with known displacements and reducing restrictions on measurement conditions. Furthermore, based on the fixed constraints of the surround-view camera array and 3D laser scanner and the constructed dynamic control field, the measurement platform is positioned and positioned with high precision for each inspection state, thereby unifying the measurement platform into a global coordinate system and effectively avoiding the inherent drawback of large cumulative errors in traditional 3D modeling methods. The high-resolution surround-view image is then mapped into 3D point cloud data in the global coordinate system, achieving high-resolution and high-precision 3D holographic stitching of the tunnel annulus in the global coordinate system, achieving both global and local high resolution and high precision. Compared to traditional technologies, this method significantly improves the accuracy of tunnel 3D hologram acquisition and allows for arbitrary perspective projection of the tunnel without introducing prior stitching model assumptions, such as obtaining a high-precision 2D panorama. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 1 is a flow chart of a method for stitching three-dimensional holographic images in a tunnel using a panoramic array camera network in one embodiment;

[0022] Figure 2 Schematic diagram of the composition of a panoramic camera array in one embodiment;

[0023] Figure 3 Schematic diagram of the relative position relationship of the measurement platform in one embodiment;

[0024] Figure 4 A schematic diagram of a dynamic networking of a reference transfer camera array in one embodiment;

[0025] Figure 5 A schematic diagram of positioning and determining the attitude of a panoramic camera array platform in one embodiment;

[0026] Figure 6 Schematic diagram of the application process of a tunnel 3D holographic stitching method using a panoramic array camera network in one embodiment;

[0027] Figure 7Schematic diagram of the module framework of a tunnel 3D holographic stitching device using a panoramic array camera network in one embodiment. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the description of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0029] It should be noted that, when referred to in this document as an "embodiment", it means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The display of this phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It will be understood by those skilled in the art that the embodiments described herein may be combined with other embodiments. The term "and / or" used herein refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0030] The following describes the implementation of the present invention in detail with reference to the accompanying drawings in the embodiments of the present invention.

[0031] 3D holographic modeling is a relatively mature measurement technology, encompassing optical measurement, photogrammetry, and computer vision. It offers advantages such as high precision, long-distance measurement, and non-contact measurement. The camera calibration methods discussed in this specification can be specifically implemented in the prior art, such as those in patents CN202410665168.8, CN202111455918.1, or CN202410261618.7.

[0032] To address the challenge of stitching high-resolution, three-dimensional panoramic views of large tunnels, the panoramic camera array mounted on the inspection vehicle (serving as a measurement platform) described in this manual consists of a surround-view camera array and a reference transfer camera array. The reference transfer camera array, consisting of two cameras (front and rear), is responsible for transmitting control information from both ends of the tunnel to the measurement area. The surround-view camera is responsible for acquiring high-resolution surround-view images of the tunnel at each inspection location. The surround-view camera array consists of two or more high-resolution cameras, each with a certain overlap in their fields of view. During the inspection process, the inspection vehicle ensures that the fields of view of each camera in the surround-view camera array overlap at adjacent inspection locations. The 3D laser scanner is responsible for collecting high-precision, dense 3D point cloud data. After completing a patrol inspection, the system uses the image keypoints in the high-resolution surround view images of the surround view camera array at adjacent patrol locations, the image keypoints, and the constraints of the 3D point cloud keypoints in the point cloud data. Combined with the fixed constraints of the datum transfer camera array and the 3D laser scanner, a joint adjustment of the datum transfer camera array and the 3D laser scanner is performed. The 3D displacement of key measurement points within the survey area relative to the global coordinate system is rapidly estimated under the condition of limited peripheral control information, thereby constructing a high-precision dynamic control field with known positions. Finally, based on this constructed dynamic control field, the position of the measurement platform at adjacent patrol locations relative to the global coordinate system is rapidly estimated, enabling rapid splicing of the 3D point cloud. Furthermore, combined with the fixed constraints of the surround view camera and the 3D laser scanner, the surround view camera texture is rapidly mapped, resulting in a global, high-resolution 3D holographic image of the tunnel.

[0033] Adjustment is a mathematical processing method based on statistics and error theory. Its core purpose is to eliminate or reduce the impact of various errors in measurement data. In essence, it is to analyze redundant observation data, reasonably distribute errors, and make the measurement results meet certain accuracy requirements and logical consistency, thereby obtaining more reliable and reasonable measurement results and providing accurate data support for tunnel deformation analysis and safety assessment.

[0034] In one embodiment, Figure 1 As shown, a tunnel 3D holographic stitching method using a panoramic array camera network is provided, which may include the following processing steps S12 to S20:

[0035] S12, offline calibration of the panoramic camera array and 3D laser scanner on the inspection vehicle; the panoramic camera array includes a surround-view camera array and a reference transfer camera array, the reference transfer camera array includes a fixedly mounted forward-view camera and a rear-view camera, which are used to transmit control information from both ends of the tunnel to the measurement area, and the surround-view camera array includes at least two high-resolution cameras with overlapping fields of view, which are used to obtain high-resolution surround-view images of the tunnel at each inspection location;

[0036] S14, using an inspection vehicle to move the panoramic camera array and the 3D laser scanner to each inspection location, using the surround-view camera array to obtain a high-resolution surround view image of the tunnel at each inspection location, using the reference transfer camera array to take photos and extract the center coordinates of the measurement points in the measurement area, and using the 3D laser scanner to collect dense 3D point cloud data in the measurement area;

[0037] S16, based on the center coordinates of the measurement points in the measurement area and the control information at both ends of the tunnel, the camera array and the 3D laser scanner at different inspection positions are jointly adjusted to construct a dynamic control field with known displacement;

[0038] S18, based on the dynamic control field, using the surround-view camera array and three or more dynamic control points observed by the 3D laser scanner, estimating the position of the measurement platform relative to the global coordinate system, and then stitching the 3D point cloud obtained by the 3D laser scanner into the global coordinate system;

[0039] S20, mapping the high-resolution surround view image to three-dimensional point cloud data in a global coordinate system according to the fixed connection constraint between the surround view camera array and the three-dimensional laser scanner, thereby obtaining a global high-resolution three-dimensional holographic image of the tunnel.

[0040] It is understandable that Figure 2 As shown, the reference transfer camera array 001 in the panoramic camera array is composed of two fixedly installed forward and rear-view cameras. Through the form of adjustment of the reference transfer camera array 001 and the three-dimensional laser scanner 004 (the inspection vehicle is equivalent to a camera measuring station when it is in each inspection position), the reference transfer camera array 001 and the three-dimensional laser scanner 004 are jointly adjusted when the inspection vehicle is in multiple inspection positions. Based on the image homonymous points, the image homonymous points and the three-dimensional point cloud homonymous points constraints in the point cloud data, combined with the fixed constraints of the reference transfer camera array and the three-dimensional laser scanner, the displacement of the inspection position and the reference transfer points in the measurement area are synchronously solved under the condition of peripheral limited control information, and the measuring points with known positions form a dynamic control field.

[0041] Surround view camera array 002 within the panoramic camera array can be composed of two or more high-resolution cameras, with their fields of view overlapping to provide high-resolution surround coverage imaging of a local section of tunnel 003 (including dynamic control point information within the field of view). Based on the three or more dynamic control points within the field of view of the high-resolution surround view image and the scanning area of ​​3D laser scanner 004, the position parameters of the measurement platform at the inspection location relative to the global coordinate system are calculated, thereby unifying the local high-resolution surround view image of tunnel 003 and the local high-precision dense 3D point cloud into the global coordinate system. 3D laser scanner 004 is responsible for scanning and acquiring high-precision dense 3D point cloud data within the measurement area.

[0042] The 3D holographic stitching method combines the rigid coupling constraints of the surround-view camera array 002 and the 3D laser scanner 004 to achieve rapid mapping of the surround-view camera texture, thereby generating local high-resolution 3D holographic images. The measurement platform moves with the inspection vehicle, enabling high-resolution 3D holographic stitching of large-scale tunnel structures. The coordinates of the peripheral control information and the initial coordinates of the dynamic control field can be obtained using a total station or other existing high-precision 3D measurement equipment. The dynamic control field refers to 3D measurement points with known 3D displacements, which are obtained through adjustment measurements of the datum transfer camera array 001 in the panoramic camera array and the 3D laser scanner 004.

[0043] Specifically, after completing an inspection, first, based on the image homonymous points, the image homonymous points and the three-dimensional point cloud homonymous points in the point cloud data, combined with the fixed connection constraints of the reference transfer camera array and the three-dimensional laser scanner, the reference transfer camera array 001 and the three-dimensional laser scanner 004 in the panoramic camera array at different inspection positions are jointly adjusted, thereby constructing a measurement link, realizing the transmission of control information from both ends of the tunnel 003 to the measurement area, measuring the displacement of the measurement points in the measurement area relative to the initial state, and then constructing a dynamic control field with known displacement; then, based on the three or more dynamic control points observed by the surround view camera array and the three-dimensional laser scanner 004 in the panoramic camera array, the position and posture of the measurement platform relative to the global coordinate are quickly estimated, and the three-dimensional point cloud scanned by the three-dimensional laser scanner 004 is spliced ​​into the global coordinate system; finally, according to the fixed connection constraints between the surround view camera array 002 and the three-dimensional laser scanner 004, the local high-resolution surround view image captured by the surround view camera array 002 is mapped to the three-dimensional point cloud data obtained by the three-dimensional laser scanner 004, thereby realizing three-dimensional modeling of the entire inspection area, and thus obtaining a global high-resolution three-dimensional holographic image.

[0044] The aforementioned panoramic array camera network-based tunnel 3D holographic stitching method utilizes a panoramic camera array and 3D laser scanner installed on an inspection vehicle. The reference transfer camera array and 3D laser scanner within the method move with the inspection vehicle to different inspection locations for adjustment. This method achieves high-precision transmission of peripheral control information from both ends of the tunnel to the measurement area, thereby constructing a dynamic control field with known displacements and reducing restrictions on measurement conditions. Furthermore, based on the fixed constraints of the surround-view camera array and 3D laser scanner and the constructed dynamic control field, the measurement platform is accurately positioned and determined for each inspection state, thereby unifying the measurement platform into a global coordinate system and effectively avoiding the inherent drawback of large cumulative errors in traditional 3D modeling methods. The high-resolution surround-view image is then mapped to 3D point cloud data in the global coordinate system, achieving high-resolution and high-precision 3D holographic stitching of the tunnel annulus in the global coordinate system, achieving both global and local high resolution and accuracy. Compared to traditional technologies, this method significantly improves the accuracy of tunnel 3D hologram acquisition and allows for arbitrary perspective projection of the tunnel without introducing prior stitching model assumptions, such as obtaining a high-precision 2D panorama.

[0045] Furthermore, a dynamic control field is constructed based on the joint adjustment of the reference transfer camera array and the 3D laser scanner in the panoramic camera array:

[0046] Existing path planning software can be used to control the panoramic camera array and 3D laser scanner on the inspection vehicle to repeatedly capture images at designated inspection locations. However, due to the enclosed environment of the tunnel, errors can occur in the positioning and attitude of the measurement platform even when the inspection vehicle is parked. To meet the requirements of high-precision measurement, this specification develops a high-precision robust estimation method for the relative position and attitude of the moving platform of the reference transfer camera array and 3D laser scanner.

[0047] like Figure 3 As shown, the relative position of the measurement platform refers to the current moment t 1Platform coordinate system Relative to the initial time t 0 platform coordinate system Posture (Use rigid body transformation matrix to represent pose). For space point , t 1 moment and t The imaging process at time 0 can be described as:

[0048] (1)

[0049] in, express t The depth factor of the space at time 0, express tThe depth factor of the spatial point at time 1, express t The homogeneous coordinates of the image point corresponding to the spatial point at time 0, express t The homogeneous coordinates of the image point corresponding to the spatial point at time 1, represents the intrinsic parameter matrix, represents three rows and one column of zero vectors, Represents the pose transformation from the measurement platform to the camera coordinate system, Represents the world coordinate system arrive t Pose transformation of the platform coordinate system at time 0.

[0050] In one embodiment, the control information at both ends of the tunnel is constructed by a total station or other coordinate precision measurement equipment.

[0051] It is understandable that in actual engineering measurements, a total station can be used to construct a control field (i.e., control information at both ends of the tunnel) at both ends of the tunnel. If the total station is leveled, the origin of the platform coordinate system can be built at the center of the measuring station, and the coordinate axes of the platform coordinate system are parallel to the coordinate axes of the total station. In this way, subsequent measurement results are based on the total station coordinate system. When the intrinsic parameters of the camera are known, the existing PnP (Perspective-n-Point, a method for estimating camera pose) or NPnP (Non-perspective PnP, a method for multi-camera pose estimation) need to first calculate the world coordinate system separately. (Control Coordinate System) to t 1 moment and t Transformation of the platform coordinate system at time 0 and Then solve the relative position of the measurement platform , this method belongs to indirect relative pose estimation.

[0052] Different from the existing design ideas, this embodiment constructs a relative pose estimation differential model to transform the indirect relative pose estimation problem into a direct pose estimation problem. In actual engineering measurement, the control point is far away from the measurement area, and the change of the depth factor of the spatial point caused by the shaking of the measurement platform is relatively small. Therefore, the depth factor invariance assumption can be introduced, that is, = , so the relative pose estimation differential model can be obtained as follows:

[0053] (2)

[0054] in, represents the identity matrix with 4 rows and 4 columns, express tPlatform coordinate system at time 0 Because the inspection vehicle's platform position and posture change little at the two retake positions, it can be described in exponential form. as follows:

[0055] (3)

[0056] in, represents the velocity and angular velocity parameters, Represents the Lie algebra basis matrix of rigid body transformation.

[0057] For large engineering structures such as tunnels, control points (PO) are usually selected from stable structures far away from the survey area, and multi-level (camera) station transfer measurement is required to transfer the control point information to the survey area. Figure 4 As shown, the reference transfer camera array and the three-dimensional laser scanner in the panoramic camera array at different positions (such as position 1 to position 3) of the inspection vehicle can be regarded as adjustment. For adjacent inspection positions, the common field of view area between the reference transfer camera arrays in the panoramic camera array can establish a connection between the reference transfer camera arrays at different inspection positions. In this way, adjustment can be achieved. Figure 4 In , P0 to P2 represent point sets respectively, and each point set includes multiple measuring points.

[0058] Relative to the initial state, the target point in the measurement area may produce a three-dimensional displacement, so the relative measurement model described by formula (2) can be modified as follows:

[0059] (4)

[0060] Among them, the corner mark T represents transpose, is the displacement of the target point in the initial platform coordinate system. If the point is a control point, then .

[0061] Similarly, for 3D laser scanners there are:

[0062] (5)

[0063] in, The target point at the initial moment t 0 coordinate system of 3D laser scanner L The location information below, is the platform coordinate system at the initial position B Coordinate system with 3D laser scanner L The rigid body transformation matrix, For the target point t 1. The coordinate system of the 3D laser scanner at this momentL The location information below, W For the world coordinate system, building a dynamic control field means building transformations under three coordinate systems to ensure higher accuracy.

[0064] The differential measurement model can be further constructed as follows:

[0065] (6)

[0066] Considering the three-dimensional displacement of the measuring point, a measurement equation similar to formula (4) can be further constructed as follows:

[0067] (7)

[0068] Equations (4) and (7) are the measurement equations for the joint adjustment of the datum transfer camera array and the 3D laser scanner.

[0069] It can be understood that the observation constraints that can be used in the dynamic network formed by the panoramic camera array and the three-dimensional laser scanner due to the movement of the inspection vehicle can be divided into two categories: one is that the different target points observed by the transfer camera array in the panoramic camera array at the same inspection position and the different target points observed by the three-dimensional laser scanner correspond to the same relative pose parameters; the other is that the same target point observed by the transfer camera array in the panoramic camera array and the three-dimensional laser scanner at different inspection positions corresponds to the same displacement parameters.

[0070] In one embodiment, the surround-view camera array includes at least three high-resolution cameras. It is understood that a surround-view camera array composed of three or more high-resolution cameras can better ensure resolution, improve imaging quality, and promote further improvement in the resolution and accuracy of three-dimensional holographic stitching.

[0071] In one embodiment, in the process of estimating the position and posture of the measurement platform relative to the global coordinate system, the cost equations of the surround-view camera array and the three-dimensional laser scanner in the global coordinate system are established respectively, and then solved using null space analysis to obtain the rotation matrix and translation vector of the measurement platform relative to the global coordinate system. The rotation matrix and translation vector are iteratively optimized and solved using a nonlinear optimization error function to complete the positioning and posture determination of the measurement platform relative to the global coordinate system.

[0072] Platform positioning and attitude determination based on the surround-view camera array and 3D laser scanner within the panoramic camera array: Joint adjustment using the fiducial transfer camera array within the panoramic camera array and the 3D laser scanner transfers peripheral control point information to the survey area, measuring the displacements of these points within the survey area. These points within the survey area with known displacements can be used as dynamic control points to construct a dynamic control field. This dynamic control field enables high-precision positioning and attitude determination of inspection vehicles at each inspection location. Given three point correspondences (the sum of the dynamic control points observed by the surround-view camera array), the position and attitude of the measurement platform relative to the global coordinate system can be linearly determined using the NPnP method, achieving platform positioning and attitude determination. Furthermore, by combining the high-precision local coordinate data of the dynamic control points measured by the 3D laser scanner, even higher-precision positioning and attitude determination can be achieved by given three point correspondences (the sum of the dynamic control points observed by the surround-view camera array and the 3D laser scanner).

[0073] Specifically, the imaging model of the camera array system composed of multiple surround-view cameras can be modeled using a non-perspective generalized camera system, such as Figure 5 As shown, P 1 to P 4 represent each dynamic control point, P 5 is a point in space, C 1 to C 4 represent the surround view cameras, represents the bias of the 3D laser scanner in a multi-camera system, O B represents the platform coordinate system, O w represents the global coordinate system, R , T They represent the rotation and translation from the global coordinate system to the platform coordinate system, corresponding to the posture and position of the platform in the global coordinate system. In this model, the relationship between the 3D coordinates and their corresponding image point coordinates can be expressed as:

[0074] (8)

[0075] in, , , Represents the normalized direction vector from the camera to the 3D point, represents the bias of the camera in the multi-camera system, and They represent the rotation matrix and translation vector from the multi-camera system coordinate system to the camera coordinate system respectively. Represents the intrinsic parameter matrix of the camera. The non-perspective generalized camera model is as follows Figure 2 shown.

[0076] forn The 2D-3D point correspondences are obtained from formula (8):

[0077] (9)

[0078] Rewrite formula (9) as:

[0079] (10)

[0080] Among them, each matrix , , , .

[0081] For the case where the control points are coplanar, the default coordinate of the 3D point in the Z direction is 0. Substituting it into equation (10) yields the rotation matrix The third column is eliminated. Therefore, in the configuration where the control points are coplanar, the matrix 、 and The dimension remains unchanged, the matrix becomes Matrix, Matrix becomes a 6-dimensional vector.

[0082] If the given matrix , the matrix can be solved by the least squares method :

[0083] (11)

[0084] in, express The pseudo-inverse of Substituting formula (11) into formula (10) yields the equation The linear equations for :

[0085] (12)

[0086] in, , .

[0087] Due to the presence of noise, Equation (12) cannot be satisfied in general, so it is converted into a least squares problem by minimizing the sum of squares of the errors:

[0088] (13)

[0089] In order to facilitate global optimization by solving polynomial systems, the rotation matrix is ​​represented by the unit quaternion ( a , b , c , d)express:

[0090] (14)

[0091] in, , substituting formula (14) into formula (13) yields the cost equation:

[0092] (15)

[0093] in, , M It's about 2 n ×11 matrix.

[0094] Similarly, for a 3D laser scanner, the 3D coordinates in the global coordinate system and the corresponding laser scanning point coordinates can be written as:

[0095] (16)

[0096] in, , and They represent the rotation matrix and translation vector from the measurement platform coordinate system to the 3D laser scanner coordinate system respectively.

[0097] Combining formulas (8) and (16) can also construct the cost equation in the form of formula (15).

[0098] The cost equation shown in formula (15) can be solved using null space analysis to obtain the unit quaternion corresponding to the rotation of the measurement platform relative to the global coordinate system. Substituting it into formula (14) can obtain the rotation matrix of the measurement platform relative to the global coordinate system, and further substituting it into formula (8) can obtain the translation vector of the measurement platform relative to the global coordinate system. Based on the linear analytical solution, a nonlinear optimization error function can be further constructed to iteratively optimize the rotation matrix and translation vector, thereby achieving high-precision positioning and attitude determination of the measurement platform.

[0099] Finally, the high-resolution global 3D holographic stitching process based on the surround-view camera array and 3D laser scanner in the panoramic camera array can be as follows Figure 6 As shown in the figure, in actual application, a panoramic camera array and a 3D (laser) scanner are first installed on the inspection vehicle to provide Figure 2The field of view is shown in Figure 1. Offline calibration is then performed, including calibration of the intrinsic and distortion parameters of the panoramic camera array, as well as the extrinsic parameters of the panoramic camera array and the 3D laser scanner. This can be accomplished directly using existing camera calibration methods in the field. The inspection vehicle then moves to each inspection location in the tunnel to collect data, including data collection and preprocessing of high-resolution surround view images, control point information, and high-precision dense 3D point cloud data. The high-resolution surround view images are then aligned with the high-precision dense 3D point cloud data. The improved dynamic control field construction and platform positioning and attitude determination described above are then performed. Offline measurement preparation is used to construct the peripheral control field and obtain the initial coordinates of the measurement points within the measurement area. The first inspection baseline transfer camera array is used to capture images and extract the center coordinates of the measurement points. The first inspection 3D laser scanner is used to measure and extract the 3D coordinates of the measurement point centers, providing the necessary data for constructing the dynamic control field with known displacements. On this basis, based on the multi-camera fixed connection constraints of the surround-view camera array, the fixed connection constraints between the surround-view camera array and the three-dimensional laser scanner, and the constructed dynamic control field, high-precision positioning and attitude determination are performed for each inspection state of the inspection vehicle, thereby unifying the measurement platform into the global coordinate system; and then, based on the fixed connection constraints between the surround-view camera array and the three-dimensional laser scanner, local high-resolution three-dimensional texture mapping is achieved, thereby realizing high-resolution and high-precision three-dimensional holographic stitching of the tunnel annulus in the global coordinate system, and finally obtaining a three-dimensional hologram of the tunnel, which greatly improves the acquisition accuracy of the tunnel three-dimensional hologram and can project the tunnel from any perspective.

[0100] It should be noted that the data processing part of the above-mentioned panoramic array camera networking tunnel three-dimensional holographic stitching method can be partially or completely executed on the computer system on the inspection vehicle, or the collected data can be uploaded to the tunnel monitoring center server or cloud server to perform corresponding calculation processing. The specific execution method can be flexibly selected according to the conditions and usage requirements in the actual application scenario.

[0101] It should be understood that although Figure 1 and Figure 6 The steps in the diagram are shown in the order indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Figure 1 and Figure 6 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0102] In one embodiment, Figure 7 As shown, a tunnel 3D holographic stitching device 100 using a panoramic array camera network is provided. The device may include an offline calibration module 11, an inspection and acquisition module 13, a control field construction module 15, a positioning and attitude determination module 17, and a holographic stitching module 19. The offline calibration module 11 is used to perform offline calibration on the panoramic camera array and 3D laser scanner on the inspection vehicle. The panoramic camera array includes a surround view camera array and a reference transfer camera array. The reference transfer camera array includes fixedly mounted forward and rear view cameras, which are used to transmit control information from both ends of the tunnel to the survey area. The surround view camera array includes at least two high-resolution cameras, each with overlapping fields of view. The inspection and acquisition module 13 is used to move the panoramic camera array and 3D laser scanner to each inspection location via the inspection vehicle. The surround view camera array is used to obtain a high-resolution surround view image of the tunnel at each inspection location. The reference transfer camera array is used to take photos and extract the center coordinates of measurement points within the survey area. The 3D laser scanner is used to collect dense 3D point cloud data within the survey area. The control field construction module 15 is used to transfer the different inspection position references to the camera array and 3D laser scanner for joint adjustment based on the center coordinates of the measurement points within the survey area and the control information at both ends of the tunnel, thereby constructing a dynamic control field with known displacements. The positioning and attitude determination module 17 is used to estimate the position of the measurement platform relative to the global coordinate system based on the dynamic control field and utilize three or more dynamic control points observed by the surround-view camera array and the 3D laser scanner. The 3D point cloud scanned by the 3D laser scanner is then stitched into the global coordinate system. The holographic stitching module 19 is used to map the high-resolution surround view image to 3D point cloud data in the global coordinate system based on the fixed connection constraints between the surround-view camera array and the 3D laser scanner, thereby obtaining a global, high-resolution 3D holographic image of the tunnel.

[0103] The panoramic array camera network-based tunnel 3D holographic stitching device 100 utilizes a panoramic camera array and 3D laser scanner mounted on an inspection vehicle. The reference transfer camera array and 3D laser scanner within the device move with the inspection vehicle to different inspection locations for adjustment, enabling high-precision transmission of peripheral control information from both ends of the tunnel to the measurement area. This constructs a dynamic control field with known displacements, reducing restrictions on measurement conditions. Furthermore, based on the fixed constraints of the surround-view camera array and 3D laser scanner and the constructed dynamic control field, high-precision positioning and attitude determination are performed for each inspection state of the inspection vehicle, thereby unifying the measurement platform into a global coordinate system and effectively avoiding the inherent drawback of large cumulative errors in traditional 3D modeling methods. The high-resolution surround-view image is then mapped into 3D point cloud data in the global coordinate system, achieving high-resolution and high-precision 3D holographic stitching of the tunnel annulus in the global coordinate system, achieving both global and local high resolution and high precision. Compared to traditional technologies, this device significantly improves the accuracy of tunnel 3D hologram acquisition and allows for arbitrary tunnel projection without introducing prior stitching model assumptions, such as obtaining a high-precision 2D panorama.

[0104] In one embodiment, the control information at both ends of the tunnel is constructed by a total station or other coordinate precision measurement equipment.

[0105] In one embodiment, in the process of estimating the position and posture of the measurement platform relative to the global coordinate system, the cost equations of the surround-view camera array and the three-dimensional laser scanner in the global coordinate system are established respectively, and then solved using null space analysis to obtain the rotation matrix and translation vector of the measurement platform relative to the global coordinate system. The rotation matrix and translation vector are iteratively optimized and solved using a nonlinear optimization error function to complete the positioning and posture determination of the measurement platform relative to the global coordinate system.

[0106] In one embodiment, the surround-view camera array includes no less than three high-resolution cameras.

[0107] It is understood that for the specific definition of the tunnel 3D holographic stitching device 100 for the panoramic array camera network, please refer to the corresponding definition of the tunnel 3D holographic stitching method for the panoramic array camera network above, and no further details will be given here. The various modules in the above-mentioned tunnel 3D holographic stitching device 100 for the panoramic array camera network can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0108] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus DRAM (RDRAM), and DDR DRAM.

[0109] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0110] The above embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of the present invention.

Claims

1. A tunnel 3D holographic stitching method using a panoramic array camera network, characterized in that: Including steps: Offline calibration is performed on the panoramic camera array and 3D laser scanner on the inspection vehicle; the panoramic camera array includes a surround-view camera array and a reference transfer camera array. The reference transfer camera array includes a rigidly mounted forward-view camera and a rear-view camera, which are used to transmit control information from both ends of the tunnel to the measurement area. The surround-view camera array includes at least two high-resolution cameras with overlapping fields of view. The inspection vehicle moves the panoramic camera array and the three-dimensional laser scanner to each inspection location, uses the surround-view camera array to obtain a high-resolution surround view image of the tunnel at each inspection location, uses the reference transfer camera array to take pictures and extract the center coordinates of the measurement points in the measurement area, and uses the three-dimensional laser scanner to collect dense three-dimensional point cloud data in the measurement area; Based on the center coordinates of the measurement points in the measurement area and the control information at both ends of the tunnel, the reference transfer camera array and the 3D laser scanner at different inspection positions are jointly adjusted to construct a dynamic control field with known displacement; Based on the dynamic control field, using the surround-view camera array and three or more dynamic control points observed by the 3D laser scanner, the position of the measurement platform relative to the global coordinate is estimated, and then the 3D point cloud scanned by the 3D laser scanner is stitched into the global coordinate system; According to the fixed connection constraint between the surround-view camera array and the 3D laser scanner, the high-resolution surround-view image is mapped to 3D point cloud data in the global coordinate system to obtain a global high-resolution 3D holographic image of the tunnel.

2. The tunnel 3D holographic stitching method of panoramic array camera networking according to claim 1, characterized in that: The control information at both ends of the tunnel is constructed through total stations or coordinate precision measurement equipment.

3. The tunnel 3D holographic stitching method of panoramic array camera networking according to claim 1 or 2, characterized in that: In the process of estimating the position and posture of the measurement platform relative to the global coordinate system, the cost equations of the surround-view camera array and the three-dimensional laser scanner in the global coordinate system are established respectively, and then solved using null space analysis to obtain the rotation matrix and translation vector of the measurement platform relative to the global coordinate system. The rotation matrix and translation vector are iteratively optimized and solved using a nonlinear optimization error function to complete the positioning and posture determination of the measurement platform relative to the global coordinate system.

4. The method for 3D holographic stitching of a tunnel using a panoramic array camera network according to claim 3, wherein: The surround-view camera array includes no less than three high-resolution cameras.

5. A tunnel 3D holographic splicing device using a panoramic array camera network, characterized in that: include: Offline calibration module, used to perform offline calibration on the panoramic camera array and 3D laser scanner on the inspection vehicle; The panoramic camera array includes a surround-view camera array and a reference transfer camera array. The reference transfer camera array includes a fixedly mounted forward-view camera and a rear-view camera for transmitting control information from both ends of the tunnel to the measurement area. The surround-view camera array includes at least two high-resolution cameras with overlapping fields of view. An inspection acquisition module is configured to move the panoramic camera array and 3D laser scanner to each inspection location via the inspection vehicle, acquire a high-resolution surround view image of the tunnel at each inspection location using the surround view camera array, take photos using the reference transfer camera array and extract the center coordinates of measurement points within the measurement area, and acquire dense 3D point cloud data within the measurement area using the 3D laser scanner; A control field construction module is used to jointly adjust the reference transfer camera array and the 3D laser scanner at different inspection positions based on the center coordinates of the measurement points in the measurement area and the control information at both ends of the tunnel to construct a dynamic control field with known displacement; A positioning and attitude determination module is used to estimate the position of the measurement platform relative to the global coordinate system based on the dynamic control field and three or more dynamic control points observed by the surround-view camera array and the 3D laser scanner, and then stitch the 3D point cloud scanned by the 3D laser scanner into the global coordinate system; The holographic stitching module is used to map the high-resolution surround view image to three-dimensional point cloud data in the global coordinate system based on the fixed connection constraints between the surround view camera array and the three-dimensional laser scanner, so as to obtain a global high-resolution three-dimensional holographic image of the tunnel.

6. The tunnel 3D holographic splicing device of panoramic array camera networking according to claim 5, characterized in that: The control information at both ends of the tunnel is constructed through total stations or coordinate precision measurement equipment.

7. The tunnel 3D holographic stitching device of the panoramic array camera network according to claim 5 or 6, characterized in that: In the process of estimating the position and posture of the measurement platform relative to the global coordinate system, the cost equations of the surround-view camera array and the three-dimensional laser scanner in the global coordinate system are established respectively, and then solved using null space analysis to obtain the rotation matrix and translation vector of the measurement platform relative to the global coordinate system. The rotation matrix and translation vector are iteratively optimized and solved using a nonlinear optimization error function to complete the positioning and posture determination of the measurement platform relative to the global coordinate system.

8. The tunnel 3D holographic stitching device of panoramic array camera networking according to claim 7, characterized in that: The surround-view camera array includes no less than three high-resolution cameras.

Images