Tunnel disease global positioning method and system fusing high-precision constraints of a mechanical arm

Abstract

The application relates to a tunnel disease global positioning method and system fusing high-precision constraints of a mechanical arm, and relates to the technical field of urban rail transit infrastructure detection. The method comprises the following steps: constructing a rail inspection platform comprising an intelligent rail inspection vehicle, a panoramic camera array, a three-dimensional laser scanner, a mechanical arm system and a combined navigation system; establishing a joint adjustment model for the rephotographing process of measuring points at adjacent time points, and constructing a dynamic control field; performing synchronous observation on the known measuring points, constructing a weighted joint cost equation, and solving the positioning and pose of the rail inspection platform in a global coordinate system to generate a three-dimensional model of the tunnel; fusing the observation data of the panoramic camera array and the three-dimensional laser scanner based on the three-dimensional model, and obtaining three-dimensional attribute information of the disease; guiding the rail inspection platform to perform disease inspection based on the three-dimensional attribute information of the disease and the normal vector of the wall surface, and obtaining disease physical parameters and disease point clouds; and performing ICP registration on the disease point clouds and the three-dimensional model and performing disease correction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of urban rail transit infrastructure detection, and in particular to a tunnel disease global positioning method and system fusing high-precision constraints of a mechanical arm. BACKGROUND

[0002] With the continuous progress of science and technology, image recognition and three-dimensional modeling technology have made significant development, and track inspection vehicles are increasingly widely used in tunnel inspection. In the inspection of large and narrow structures such as subway tunnels, achieving high-precision positioning and orientation of intelligent track inspection and three-dimensional reconstruction of tunnel holographic images is an important basis for carrying out tunnel structure detection and disease diagnosis.

[0003] In the prior art, some solutions to related problems through specific ways have also appeared. For example, Chinese application publication No. CN120634851A discloses a tunnel three-dimensional holographic splicing method and device of panoramic array camera networking, which specifically adopts a way of fixedly connecting a panoramic camera array and a three-dimensional laser scanner to constrain, constructs a dynamic control field for positioning and orientation, and realizes three-dimensional holographic splicing based on the control field. However, this solution has certain limitations: on the one hand, only relying on such “non-contact measurement” devices as the panoramic camera array and the three-dimensional laser scanner is easy to be disturbed by the environment, thereby causing positioning errors. On the other hand, the constraints provided by the panoramic camera array and the three-dimensional laser scanner are relatively weak. For example, when the inspection vehicle encounters vibration and yaw during movement, there will be a small time drift or registration error between the data collected by the camera and the laser scanner. As the inspection continues, these accumulated errors may affect the stability of the overall modeling.

[0004] For another example, Chinese application publication No. CN106680290A discloses a multi-functional detection vehicle for narrow spaces, which realizes automatic inspection capability by carrying a mechanical arm, but its positioning process excessively relies on sensors for point recognition, and the collected data is difficult to effectively apply to three-dimensional modeling. For the acquired tunnel disease images, only preliminary identification can be performed through related algorithms, and three-dimensional spatial coordinates cannot be assigned to related diseases, so that the position information of the diseases in space is difficult to accurately determine.

[0005] It can be seen that the current track inspection vehicle system mostly focuses on the front-end processing capability of image acquisition and recognition algorithm, although it can realize automatic detection and image labeling of diseases such as cracks, but these results often only stay at the "visual level". Often lack of measurement capability of real scale, geometric shape and depth characteristics of disease targets in physical space. Especially in the complex tunnel structure environment, the image recognition result is difficult to accurately map to a unified three-dimensional space coordinate system, and cannot provide reliable spatial guidance information for subsequent mechanical arm operation. This "recognition and execution disconnection" condition makes it difficult for the track inspection vehicle to undertake higher level structure health detection and active intervention tasks.

[0006] Therefore, it is urgent to propose a new tunnel disease detection scheme to achieve the functional closed-loop upgrade from "perception recognition" to "positioning execution" to meet the actual needs of high-precision detection and effective intervention in the tunnel inspection field. SUMMARY

[0007] The technical problem to be solved by the present application is to provide a tunnel disease global positioning method and system fused with high-precision constraints of a mechanical arm, which is used to solve the problems of accurate positioning of various diseases in tunnels and acquisition of physical information related to disease void bodies.

[0008] To achieve the above-mentioned application purposes, the present application provides a tunnel disease global positioning method fused with high-precision constraints of a mechanical arm, comprising the following steps:

[0009] S1. Constructing a track inspection platform, wherein the track inspection platform comprises an intelligent track inspection vehicle, a panoramic camera array, a three-dimensional laser scanner, a mechanical arm system and a combined navigation system carried on the intelligent track inspection vehicle;

[0010] S2. Setting a global coordinate system of the tunnel, establishing a joint adjustment model of the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system based on the rephotographing process of the same measuring point at adjacent time points, and constructing a dynamic control field of the known measuring point;

[0011] S3. Based on the dynamic control field, performing synchronous observation on the known measuring point by the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system, constructing a weighted joint cost equation, and solving the positioning and pose of the track inspection platform in the global coordinate system by the least square method;

[0012] S4. Based on the positioning and pose, performing global three-dimensional holographic splicing on the observation data of the panoramic camera array and the three-dimensional laser scanner to generate a three-dimensional model of the tunnel;

[0013] S5. Based on the three-dimensional model, complementarily fusing the observation data of the panoramic camera array and the three-dimensional laser scanner, and loading the fused features into a target detection network for disease recognition to obtain three-dimensional attribute information of the disease and complete one inspection;

[0014] S6. Based on the three-dimensional attribute information of the disease and the normal vector of the wall surface where the disease is located, a secondary inspection path of the rail inspection platform is planned, the rail inspection platform is guided to perform disease inspection, and disease physical parameters and disease point clouds are obtained;

[0015] S7. The disease point cloud is registered with the three-dimensional model through ICP, the disease is corrected according to the registration result, and the secondary inspection is completed.

[0016] According to one aspect of the present application, in step S1, in the step of constructing the rail inspection platform, the panoramic camera array comprises: a surround-view camera array and a reference transfer camera array;

[0017] The surround-view camera array is used to obtain surround-view images at each inspection position in the tunnel, and has at least two surround-view cameras.

[0018] The reference transfer camera array is used to transfer control information of the global coordinate system at both ends of the tunnel to the inspection area in the tunnel, and has a forward-looking camera and a rear-view camera.

[0019] The three-dimensional laser scanner is used to obtain three-dimensional point clouds at each inspection position in the tunnel.

[0020] The mechanical arm system comprises: a multi-degree-of-freedom mechanical arm, and a range finder installed at the end of the multi-degree-of-freedom mechanical arm.

[0021] The multi-degree-of-freedom mechanical arm is used to control the pose and position of the range finder, and the range finder is used to measure the measuring points in the tunnel.

[0022] The integrated navigation system is used for navigation of the intelligent rail inspection vehicle in the tunnel, and has at least one of an IMU device, a GNSS device and a DMI device.

[0023] According to one aspect of the present application, in step S2, in the step of establishing a joint adjustment model of the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system based on the repeated shooting process of the same measuring point at adjacent time points, and constructing the dynamic control field of the known measuring point, comprising:

[0024] S21. Linear difference measurement models of the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system are respectively established based on the repeated shooting process of the same measuring point at adjacent time points;

[0025] S22. Based on the observation constraints in the dynamic network formed by the movement of the rail inspection platform, the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system, and under the condition of giving the highest confidence weight to the geometric constraints of the mechanical arm system, the three linear difference measurement models are constructed into a unified weighted least squares adjustment system, the relative pose parameters of the rail inspection platform and the displacement of the measuring point are solved, and the dynamic control field is constructed.

[0026] According to one aspect of the present application, in step S21, the linear difference measurement model of the mechanical arm system is represented as:

[0027] ;

[0028] ;

[0029] wherein, represents the linear difference measurement model of the mechanical arm system at adjacent time points , , represents the transformation matrix from the platform coordinate system of the rail inspection platform to the base coordinate system of the multi-degree-of-freedom mechanical arm in the mechanical arm system, represents the sum of the displacement of the intelligent rail inspection vehicle and the three-dimensional displacement of the measurement point itself, represents the cumulative basis function coefficient of the B-spline curve at the corresponding time, represents the Lie algebra control point basis matrix in the SE(3) space, represents the three-dimensional coordinates of the measurement point in the platform coordinate system at time point , represents the displacement of the measurement point in the platform coordinate system at time point , represents the transformation matrix from the platform coordinate system to the camera coordinate system ,

[0030] The linear difference measurement model of the panoramic camera array is represented as:

[0031] ;

[0032] wherein, represents the linear difference measurement model of the panoramic camera array at adjacent time points , , represents the spatial depth factor of the measurement point relative to the camera optical center at time point , represents the camera intrinsic parameter, represents the transformation matrix from the platform coordinate system to the camera coordinate system , represents a zero vector for matrix dimension matching;

[0033] The linear difference measurement model of the three-dimensional laser scanner is represented as:

[0034] ;

[0035] wherein, denotes the linear difference measurement model of the three-dimensional laser scanner at adjacent time points 、 , denotes the transformation matrix from the platform coordinate system to the three-dimensional laser scanner coordinate system .

[0036] According to an aspect of the present application, in step S22, the observation constraints in the dynamic network include: co-location multi-module pose consistency constraints and cross-location same point displacement consistency constraints;

[0037] The co-location multi-module pose consistency constraints are: different points observed by the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system correspond to the same relative pose parameters at the same inspection location.

[0038] The cross-location same point displacement consistency constraints are: the same point observed by the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system corresponds to the same displacement parameters at different inspection locations.

[0039] According to an aspect of the present application, in step S3, based on the dynamic control field, the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system are used to perform synchronous observation on the known points, and the step of constructing the weighted joint cost equation is represented as:

[0040] ;

[0041] ;

[0042] ;

[0043] ;

[0044] wherein, denotes the weighted joint cost equation, denotes the confidence weight of the panoramic camera array, denotes the confidence weight of the three-dimensional laser scanner, denotes the confidence weight of the mechanical arm system, and is greater than the other two confidence weights, denotes the cost equation derived from the observation equation of the panoramic camera array, denotes the cost equation derived from the observation equation of the three-dimensional laser scanner, denotes the cost equation derived from the observation equation of the mechanical arm system, Let represent the rotation matrix in the positioning and orientation determination of the track inspection platform to be solved. Let represent the translation vector in the positioning and attitude determination of the track inspection platform to be solved. Let represent the normalized direction vector from the camera to the measurement point, and subscript Indicates the serial number of the measuring point. Represents the pixel coordinates of the measurement point. Let represent the intrinsic parameter matrix of the camera, and , Represents the platform coordinate system To a single camera coordinate system The rotation matrix, Indicates the camera serial number. This represents the world coordinates of a known measurement point, i.e., its three-dimensional coordinates in the global coordinate system. Represents the platform coordinate system To a single camera coordinate system The translation vector, This represents the three-dimensional coordinates of a known measurement point acquired by a 3D laser scanner. Represents the platform coordinate system To the coordinate system of the 3D laser scanner The rotation matrix, Represents the platform coordinate system To the coordinate system of the 3D laser scanner The translation vector, This represents the three-dimensional coordinates of known measurement points acquired in the base coordinate system of a multi-degree-of-freedom robotic arm. Let represent the rotation matrix from the platform coordinate system of the track inspection platform to the base coordinate system of the multi-degree-of-freedom robotic arm, and , Let represent the translation vector from the platform coordinate system of the track inspection platform to the base coordinate system of the multi-degree-of-freedom robotic arm, and , This indicates a normalization operation.

[0045] According to one aspect of the present invention, step S4, which involves generating a three-dimensional model of the tunnel by performing global three-dimensional holographic stitching based on the positioning pose using observation data from a panoramic camera array and a three-dimensional laser scanner, includes:

[0046] The observation data from the panoramic camera array and the 3D laser scanner are acquired and preprocessed separately. The observation data includes: the panoramic RGB images acquired by the panoramic camera array and the 3D point cloud of the 3D laser scanner.

[0047] Obtain the positioning and attitude of the track inspection platform in the global coordinate system, and unify the observation data to the global coordinate system based on the positioning and attitude.

[0048] The positioning and pose determination based on the adjacent inspection positions splices the three-dimensional point cloud to obtain a global three-dimensional point cloud skeleton;

[0049] Image point color information of the surround-view RGB image corresponding to each point cloud point is extracted, and 3D texture mapping is performed on the three-dimensional point cloud skeleton to construct a three-dimensional model of the tunnel.

[0050] According to one aspect of the present application, in step S5, in the step of complementary fusion of the observation data of the panoramic camera array and the three-dimensional laser scanner based on the three-dimensional model, bidirectional complementary fusion is performed on the image points in the surround-view RGB image collected by the panoramic camera array and the point cloud points in the three-dimensional point cloud collected by the three-dimensional laser scanner to obtain fusion features, and the bidirectional complementary fusion comprises:

[0051] The three-dimensional point cloud collected by the three-dimensional laser scanner is converted into discrete point cloud voxel features, and 2D image features are extracted based on the image points in the surround-view RGB image;

[0052] The point cloud voxel features are projected to the 2D image features, neighboring image points are retrieved and fused with weights, and semantic enhanced point cloud voxel features are generated;

[0053] The point cloud voxel features are projected to generate a sparse depth map, a dense depth feature map is obtained by using a depth completion method, and after the dense depth feature map and the 2D image features are spliced, dimensional fusion is performed to generate spatial enhanced image features;

[0054] The spatial enhanced image features are lifted to the same three-dimensional space as the point cloud voxel features, and after adaptive weighted fusion with the semantic enhanced point cloud voxel features, the fusion features are obtained;

[0055] In step S5, in the step of loading the fusion features into a target detection network for disease identification and obtaining disease attribute information, the disease attribute information includes: disease number, center coordinates, length, width, recognition confidence, and collection time.

[0056] According to one aspect of the present application, in step S6, based on the three-dimensional attribute information of the disease and the normal vector of the wall surface where the disease is located, a secondary inspection path of the rail inspection platform is planned, the rail inspection platform is guided to autonomously navigate to the disease area, and the disease physical parameters and the disease point cloud are obtained, which comprises:

[0057] Based on the three-dimensional attribute information of the disease, the disease center point coordinates and the corresponding wall surface normal vector are obtained;

[0058] Based on the combined navigation system and by algorithm and PID closed-loop control, the rail inspection platform is guided to autonomously navigate to the disease area;

[0059] When the rail inspection platform reaches the working position, the mechanical arm system performs smooth wall-adhesion operation on the disease based on inverse kinematics and spatial interpolation trajectory planning, and obtains the disease physical parameters and disease point cloud;

[0060] In step S7, the disease point cloud is registered with the three-dimensional model by ICP, and the disease correction is performed according to the registration result. The registration of the disease point cloud and the three-dimensional model is based on a registration optimization function, and the registration optimization function is expressed as:

[0061] ;

[0062] Wherein, represents the best rotation transformation matrix between the disease local point cloud to be solved and the three-dimensional model point cloud, represents the best translation transformation vector between the disease local point cloud to be solved and the three-dimensional model point cloud, represents the disease point cloud collected by the smooth wall-adhesion operation, represents the disease point cloud in the three-dimensional model, represents the index of the corresponding point pair in the point cloud;

[0063] The disease center coordinates after disease correction are:

[0064] ;

[0065] Wherein, represents the corrected disease center coordinates, represents the disease center coordinates in the three-dimensional model, i.e. the disease center coordinates before correction.

[0066] To achieve the above-mentioned purposes, the present application provides a tunnel disease global positioning system integrating a mechanical arm high-precision constraint, comprising:

[0067] A rail inspection platform, comprising: an intelligent rail inspection vehicle, a panoramic camera array, a three-dimensional laser scanner, a mechanical arm system and a combined navigation system mounted on the intelligent rail inspection vehicle;

[0068] A global coordinate system construction module for setting a global coordinate system of the tunnel;

[0069] A dynamic control field construction module for establishing a joint adjustment model of the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system for the rephotographing process of the same measuring point at adjacent time points, and constructing a dynamic control field of the known measuring point;

[0070] A positioning and orientation solving module based on the dynamic control field, which uses the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system to perform synchronous observation on the known measuring point, constructs a weighted joint cost equation, and solves the positioning and orientation of the rail inspection platform in the global coordinate system by the least square method;

[0071] The holographic splicing module performs global three-dimensional holographic splicing based on the positioning pose, uses observation data of the panoramic camera array and the three-dimensional laser scanner, and generates a three-dimensional model of the tunnel;

[0072] The disease primary identification module fuses the observation data of the panoramic camera array and the three-dimensional laser scanner based on the three-dimensional model, loads the fused features into a target detection network for disease identification, and obtains three-dimensional attribute information of the disease;

[0073] The disease secondary identification module plans a secondary inspection path of the rail inspection platform based on the three-dimensional attribute information of the disease and a normal vector of a wall surface where the disease is located, guides the rail inspection platform to perform disease inspection, and obtains disease physical parameters and a disease point cloud;

[0074] The disease correction module performs ICP registration on the disease point cloud and the three-dimensional model, corrects the disease according to a registration result, and outputs a disease positioning result.

[0075] The technical effects of the present application are as follows:

[0076] According to one scheme of the present application, the scheme can realize omnidirectional, automatic, and globally high-precision three-dimensional holographic modeling of a subway tunnel and global positioning of tunnel diseases.

[0077] According to one scheme of the present application, the scheme solves the positioning problem of a weak constraint of a traditional "non-contact measurement" by introducing active wall-pasting measurement of a high-precision mechanical arm as a high-weight constraint and constructing a unified differential measurement equation group of the mechanical arm system, the panoramic camera array, and the three-dimensional laser scanner. In addition, the scheme adopts a "two-step solving" strategy and a weighted joint cost equation, gives the highest confidence to the physical measurement value of the mechanical arm system, accurately solves the platform pose, ensures the accuracy of global three-dimensional holographic splicing, and effectively avoids error accumulation.

[0078] According to one scheme of the present application, the scheme can enable an intelligent rail inspection vehicle to reach a disease detection point under the control of a combined navigation system. The mechanical arm system is unfolded and wall-pasting detection is performed on the disease point to obtain physical properties related to the disease and its void body, which facilitates further analysis of the tunnel structure. In addition, the scheme can repeatedly implement the method according to the number of introduced disease point coordinates. After completing the measurement of the last disease point, the scheme automatically returns to the end of the tunnel to realize automatic wall-pasting detection in the whole process.

[0079] According to one scheme of the present application, the scheme realizes three-dimensional reconstruction of a holographic image of a tunnel, assignment of disease coordinate information, acquisition of physical properties related to the disease and its void body, and high-precision review of three-dimensional spatial coordinates of the disease.

[0080] According to one scheme of the present application, the scheme proposes a method of image point-point cloud point bidirectional complementary fusion, which significantly improves the robust recognition ability of the model for complex disease morphology.

[0081] According to one scheme of the present application, the scheme breaks through the technical limitation of disconnection between recognition and execution, and not only can assign a preliminary three-dimensional coordinate to the disease, but also can autonomously guide the intelligent rail inspection vehicle and the mechanical arm system to accurately navigate to the target area. While the mechanical arm system acquires the key physical properties such as disease depth and volume, it also performs high-precision review and correction on the global spatial coordinates of the disease through ICP point cloud registration, truly realizing the closed-loop task chain from three-dimensional modeling, disease positioning to detection and execution, and greatly improving the precision, integrity and automation level of tunnel structure detection.

[0082] According to one scheme of the present application, the scheme introduces the active wall-pasting measurement of high-precision mechanical arm as a high-weight geometric constraint, solving the positioning problem of weak constraint of traditional non-contact measurement.

[0083] According to one scheme of the present application, the scheme constructs a tightly coupled optimization model based on the coordinate systems of various sensors, and through the conversion of the coordinate systems of the mechanical arm system, panoramic camera array and three-dimensional laser instrument, the three are unified in the platform and world coordinate systems, effectively avoiding the noise distribution distortion and precision loss caused by projecting the original measurement values to the platform coordinate system. BRIEF DESCRIPTION OF DRAWINGS

[0084] Figure 1 A step diagram of the fusion mechanical arm high-precision constraint tunnel disease global positioning method of the present application;

[0085] Figure 2 A principle diagram of establishing a joint adjustment model of the panoramic camera array, three-dimensional laser scanner and mechanical arm system of the present application;

[0086] Figure 3 An architecture diagram of using the observation data of the panoramic camera array and three-dimensional laser scanner for global three-dimensional holographic stitching of the present application;

[0087] Figure 4 A flowchart of bidirectional complementary fusion of image points in the surround view RGB image and point cloud points in the three-dimensional point cloud in the present application;

[0088] Figure 5 A disease recognition result diagram of the present application. DETAILED DESCRIPTION

[0089] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description only show some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0090] The present application will be described in detail below in conjunction with the drawings and specific embodiments. The embodiments cannot be exhaustively described here, but the embodiments of the present application are not limited to the following embodiments.

[0091] As Figure 1 shown, according to an embodiment of the present application, a fusion mechanical arm high-precision constraint tunnel disease global positioning method of the present application comprises the following steps:

[0092] S1. Constructing a track inspection platform, wherein the track inspection platform comprises: an intelligent track inspection vehicle, a panoramic camera array, a three-dimensional laser scanner, a mechanical arm system and a combined navigation system mounted on the intelligent track inspection vehicle;

[0093] S2. Setting a global coordinate system of the tunnel, establishing a joint adjustment model of the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system based on the rephotographing process of the same survey point at adjacent time points, and constructing a dynamic control field of the known survey point;

[0094] S3. Based on the dynamic control field, using the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system to perform synchronous observation on the known survey point, constructing a weighted joint cost equation, and solving the positioning and pose of the track inspection platform in the global coordinate system by the least square method;

[0095] S4. Based on the positioning and pose, using the observation data of the panoramic camera array and the three-dimensional laser scanner to perform global three-dimensional holographic stitching, and generating a three-dimensional model of the tunnel;

[0096] S5. Based on the three-dimensional model, complementarily fusing the observation data of the panoramic camera array and the three-dimensional laser scanner, and loading the fused features into a target detection network for disease identification, obtaining three-dimensional attribute information of the disease, and completing a patrol inspection;

[0097] S6. Based on the three-dimensional attribute information of the disease and the normal vector of the wall surface where the disease is located, planning a secondary patrol path of the track inspection platform, guiding the track inspection platform to perform disease patrol inspection, and obtaining disease physical parameters and disease point cloud;

[0098] S7. ICP registration of the disease point cloud and the three-dimensional model, disease correction according to the registration result, and completion of the secondary patrol inspection.

[0099] As Figure 2As shown, according to one embodiment of the present invention, in step S1, the step of constructing the track inspection platform, the panoramic camera array comprises: a surround-view camera array and a reference transfer camera array; wherein, the surround-view camera array is used to acquire surround-view images of each inspection location in the tunnel, and it has at least two surround-view cameras; the reference transfer camera array is used to transfer control information of the global coordinate system at both ends of the tunnel to the inspection area in the tunnel, and it has a front-view camera and a rear-view camera; the three-dimensional laser scanner is used to acquire the three-dimensional point cloud of each inspection location in the tunnel;

[0100] Therefore, based on the established reference transfer camera array, the robotic arm system and the 3D laser scanner can be jointly adjusted to construct a dynamic control field; the established surround view camera array is responsible for capturing high-resolution images inside the tunnel, ensuring that feature transfer and fusion can be achieved with the point cloud features of the 3D point cloud collected by the 3D laser scanner.

[0101] In this embodiment, the robotic arm system includes a multi-degree-of-freedom robotic arm and a rangefinder mounted at the end of the multi-degree-of-freedom robotic arm. The multi-degree-of-freedom robotic arm is used to control the pose and position of the rangefinder, which is used to measure points within the tunnel wall. In this embodiment, the rangefinder can actively measure high-precision feature points in the internal area and serve as a high-weight constraint. Therefore, based on the constructed dynamic control field, the intelligent track inspection vehicle can still achieve high-precision positioning and attitude determination throughout the entire process in a tunnel environment without signal. Furthermore, the robotic arm system's high-precision physical wall-hugging measurement structure can perform data reinforcement in key areas, achieving a combination of "macroscopic transmission" and "local reinforcement." This ensures the accurate alignment of the tunnel segment surface texture and the 3D point cloud in the global coordinate system, effectively suppressing the accumulation and spread of splicing errors.

[0102] In this embodiment, the integrated navigation system is used for the intelligent track inspection vehicle to navigate in the tunnel, and it has at least one of an IMU device, a GNSS device, and a DMI device; the integrated navigation system can participate in the entire process of intelligent track inspection vehicle navigation and route optimization in the tunnel.

[0103] like Figure 2 As shown, according to one embodiment of the present invention, in step S2, in the step of setting the global coordinate system of the tunnel, a total station can be used to construct a global control field at both ends of the tunnel. Specifically, by leveling the total station, the origin of the platform coordinate system of the track inspection platform can be set at the center of the total station, and the coordinate axes of the platform coordinate system of the track inspection platform are parallel to the coordinate axes of the total station. In this way, the total station coordinate system can be used as the global coordinate system to provide a global reference for subsequent measurement results.

[0104] According to an embodiment of the present application, in step S2, based on the process of repeatedly shooting the same measuring point at adjacent time points, a joint adjustment model of the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system is established, and in the step of constructing the dynamic control field of the known measuring point, the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system in the rail inspection platform can be controlled to repeatedly shoot at the specified position by means of the pre-set path planning method (such as corresponding pre-setting by using path planning software). In this process, by constructing the joint adjustment model, the active "wall measurement" of the measuring point on the tunnel wall surface by the mechanical arm system is given the highest confidence weight of geometric constraint, and the "non-contact measurement" of the panoramic camera array and the three-dimensional laser scanner is given a lower confidence weight of geometric constraint, so as to eliminate the error of positioning and orientation based on the panoramic camera array and the three-dimensional laser scanner, and realize high-precision and robust estimation of positioning and orientation. Finally, based on the joint adjustment model, a unified weighted least squares adjustment system is constructed, the relative pose parameters and the displacement of the measuring point of the rail inspection platform are solved, and the corresponding dynamic control field is constructed. Therefore, step S2 specifically includes:

[0105] S21. Linear difference measurement models of the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system are respectively established based on the process of repeatedly shooting the same measuring point at adjacent time points;

[0106] S22. Based on the observation constraints in the dynamic network formed by the movement of the rail inspection platform and the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system, and under the condition that the mechanical arm system is given the highest confidence weight of geometric constraint, the three linear difference measurement models are constructed into a unified weighted least squares adjustment system, the relative pose parameters and the displacement of the measuring point of the rail inspection platform are solved, and the dynamic control field is constructed.

[0107] According to an embodiment of the present application, in step S21, in the step of respectively establishing linear difference measurement models of the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system based on the process of repeatedly shooting the same measuring point at adjacent time points, the linear difference measurement model of the mechanical arm system is represented as:

[0108] ;

[0109] ;

[0110] wherein, represents the linear difference measurement model of the mechanical arm system at adjacent time points , represents the transformation matrix from the platform coordinate system of the rail inspection platform to the base coordinate system of the multi-degree-of-freedom mechanical arm in the mechanical arm system, ​represents the sum of the displacement of the intelligent rail inspection vehicle and the three-dimensional displacement of the measuring point itself, represents the cumulative basis function coefficient of the B-spline curve at the corresponding time, represents the Lie algebra control point basis matrix in the SE(3) space, represents the three-dimensional coordinates of the measuring point in the platform coordinate system at the time point , represents the displacement of the measuring point in the platform coordinate system at the time point , represents the summation index of the B-spline basis function;

[0111] To further illustrate the scheme, the construction process of the linear difference measurement model of the mechanical arm system is further described.

[0112] Specifically, referring to Figure 2 , the mechanical arm system is a high-precision mechanical arm system installed on the intelligent rail inspection vehicle to ensure better adaptability of the dynamic control field to the characteristic areas of the tunnel wall surface. Further, the driving mechanical arm system actively measures the same stable natural feature point (i.e., measuring point) at two time points , respectively, and considers it as a dynamic control point. The error model of the multi-degree-of-freedom mechanical arm for the dynamic control point is introduced, and is represented as:

[0113] (1)

[0114] wherein, represents the coordinates of the dynamic control point in the mechanical arm base coordinate system at the time point , represents the transformation matrix from the world coordinate system to the platform coordinate system at the time point , represents the measuring point coordinates in the world coordinate system obtained by a total station or other high-precision instrument.

[0115] Similarly, for the time point , there is:

[0116] (2)

[0117] wherein, represents the coordinates of the dynamic control point in the mechanical arm base coordinate system at the time point , represents the transformation matrix from the world coordinate system to the platform coordinate system between the two, the transformation matrix of the platform posture.

[0118] Further, in the pre-constructed global coordinate system, the aforementioned two high-precision coordinates measured independently by the mechanical arm system and , the relative pose of the platform of the track inspection platform and the displacement between the measuring points need to satisfy the following rigid body kinematics constraints:

[0119] (3)

[0120] wherein, denotes the unit matrix.

[0121] Further, assuming that the measurement interval of the mechanical arm system on the same natural feature point (i.e. measuring point) is short, i.e. the interval between the time points , is very short (i.e. less than the preset time interval), the platform position and attitude change of the track inspection platform is small, and then the exponential form can be used to describe , and is expressed as:

[0122] (4)

[0123] wherein, denotes the natural exponential function.

[0124] For large engineering structures such as tunnels, the calibration control points are usually selected from stable structures away from the measurement area. The traditional adjustment model construction requires multiple levels of camera station transmission measurements to transfer control information to the measurement area. In this process, the reference transmission camera array at different positions can be regarded as adjustment, and the public field of view area between the reference transmission camera arrays of adjacent inspection positions can establish the connection between the reference transmission camera arrays of different inspection positions. In this way, adjustment can be realized.

[0125] Based on the wall adhesion measurement function of the mechanical arm system set, the high repeatability and measurement accuracy can obtain the accurate spatial three-dimensional coordinates of the measuring points, and then the measuring points can be regarded as stable reference points introduced into the dynamic control field, which together with the image points in the image collected by the ring-view camera array and the point cloud points of the three-dimensional point cloud data collected by the three-dimensional laser scanner constitute a joint measurement constraint network, providing higher constraint redundancy for subsequent platform attitude estimation of the track inspection platform, and overcoming the inherent constraint limitations of traditional non-contact measurement.

[0126] Therefore, the linear difference measurement model of the mechanical arm system can be obtained as follows:

[0127] (5)

[0128] Considering that the three-dimensional displacement of the multi-degree-of-freedom robot arm for the measuring point and the movement of the track inspection platform in a short time are both infinitesimal quantities, the product of the two is a second-order small quantity, the linear difference measurement model of the robot arm system can be further simplified as:

[0129] (6)

[0130] wherein, represents two items, and both items are infinitesimal quantities. The former is the displacement caused by the movement of the track inspection platform, and the latter is the three-dimensional displacement of the measuring point, i.e., the three-dimensional displacement of the multi-degree-of-freedom robot arm for the measuring point.

[0131] Therefore, formula (6) can be simplified as:

[0132] (7)

[0133] In the embodiment, for the measuring point, if the measuring point is a dynamic control point, ;

[0134] Therefore, the construction of the linear difference measurement model of the robot arm system is completed.

[0135] In the embodiment, for the panoramic camera array, the linear difference measurement model of the panoramic camera array is represented as:

[0136] (8)

[0137] wherein, represents the linear difference measurement model of the panoramic camera array at adjacent time points , represents the spatial depth factor of the measuring point relative to the camera optical center at the time point , represents the camera intrinsic parameter, represents the transformation matrix from the platform coordinate system to the camera coordinate system , and represents a zero vector for matrix dimension matching.

[0138] In the embodiment, for the three-dimensional laser scanner, the linear difference measurement model of the three-dimensional laser scanner is represented as:

[0139] (9)

[0140] wherein, represents the linear difference measurement model of the three-dimensional laser scanner at adjacent time points , ​​ representing a platform coordinate system to a three-dimensional laser scanner coordinate system a transformation matrix.

[0141] Thus, based on formula (7), formula (8) and formula (9), a unified differential measurement equation group of the mechanical arm system, the panoramic camera array, the three-dimensional laser scanner and the rail inspection platform is constituted.

[0142] According to an embodiment of the present application, in the step S22, based on the observation constraints in the dynamic network formed by the movement of the rail inspection platform, the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system, the observation constraints in the dynamic network include: the same position multi-module pose consistency constraint and the cross-position same measuring point displacement consistency constraint; wherein the same position multi-module pose consistency constraint is that, under the same inspection position, different measuring points observed by the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system correspond to the same relative pose parameter; and the cross-position same measuring point displacement consistency constraint is that, under different inspection positions, the same measuring point observed by the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system correspond to the same displacement parameter.

[0143] Based on this, by fusing the three linear differential measurement models to construct a unified weighted least squares adjustment system, the peripheral control information (i.e. the global coordinate system) can be transmitted to the measurement area, the displacement of the measuring point in the measurement area is measured, the measuring point in the measurement area with known displacement can be used as a dynamic control point, and a dynamic control field is constructed. Based on the dynamic control field, high-precision positioning and pose determination of the inspection vehicle at each inspection position can be realized.

[0144] According to an embodiment of the present application, in the step S3, based on the dynamic control field, the step of using the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system to perform synchronous observation on the known measuring point, when the intelligent rail inspection vehicle stops at any inspection position, the known measuring point in the dynamic control field is synchronously observed by fusing the constructed weighted least squares adjustment system, to solve the absolute pose of the rail inspection platform relative to the global coordinate system (the measuring point (i.e. the dynamic control point) is a point in the global coordinate system calibrated by a total station or in the tunnel) with high precision , and the least squares method is used to solve, so as to realize high-precision positioning and pose determination of the rail inspection platform. Taking the known measuring point of the first camera as an example, the specific implementation of performing synchronous observation and constructing a weighted joint cost equation is further introduced.

[0145]

[0146] ​​​For high-precision robotic arm systems, the contact rangefinder mounted on its end effector can also measure points in a dynamic control field. Active physical measurements are performed. Through the forward kinematics of the multi-degree-of-freedom robotic arm, the coordinates of the base of the multi-degree-of-freedom robotic arm can be obtained. The high-precision measurement coordinates of the measuring point are then... Platform coordinate system of the on-orbit inspection platform The coordinates below are as follows:

[0147] ;

[0148] ;

[0149] in, Indicates the measuring point In the base coordinate system The measured coordinates below, This represents the positive kinematics function, which calculates the position of the robotic arm's end effector in space by inputting joint angles. This represents the joint angle vector of a multi-degree-of-freedom robotic arm. Indicates the measuring point Platform coordinate system of the on-orbit inspection platform The coordinates below, Represents the base coordinate system To platform coordinate system The rotation matrix, Represents the platform coordinate system to base coordinate system The translation vector.

[0150] Due to the measuring point Having a known three-dimensional coordinate system, therefore the platform coordinate system With global coordinate system There are:

[0151] ;

[0152] in, Indicates the measuring point In the platform coordinate system The measured coordinates below, Let represent the rotation matrix in the positioning and orientation determination of the track inspection platform to be solved. Let represent the translation vector in the positioning and attitude determination of the track inspection platform to be solved.

[0153] Here, a multi-degree-of-freedom robotic arm and dynamic control points (i.e., measurement points) can be constructed. The relational equation (i.e., the observation equation) is as follows:

[0154] ;

[0155] in, Let represent the rotation matrix from the platform coordinate system of the track inspection platform to the base coordinate system of the multi-degree-of-freedom robotic arm, and , Let represent the translation vector from the platform coordinate system of the track inspection platform to the base coordinate system of the multi-degree-of-freedom robotic arm, and .

[0156] Similarly, the observation equation for a panoramic camera array is as follows:

[0157] (10)

[0158] in, Indicates the first The spatial depth factor of each measurement point relative to the camera's optical center This represents the normalized direction vector from the camera to the measurement point (a 3D point, corresponding to a pixel in the photograph; its dimension is determined by combining the 2D dimension of the image captured by a single camera with the dimension of the corresponding point in real space). subscript Indicates the camera's serial number. Represents pixel coordinates, Let represent the intrinsic parameter matrix of the camera, and , Represents the platform coordinate system To a single camera coordinate system The rotation matrix, This represents the world coordinates of a known measurement point, i.e., its three-dimensional coordinates in the global coordinate system. Represents the platform coordinate system To a single camera coordinate system The translation vector.

[0159] Similarly, for a 3D laser scanner, in the global coordinate system Collect the 3D coordinates of several points, and thus, in the global coordinate system The following equation establishes the relationship between the 3D coordinates of the collected points and their corresponding laser scanning point coordinates:

[0160] (11)

[0161] in, This represents the three-dimensional coordinates of a known measurement point acquired by a 3D laser scanner. Represents the platform coordinate system To the coordinate system of the 3D laser scanner The rotation matrix, Represents the platform coordinate system To the coordinate system of the 3D laser scanner The translation vector.

[0162] Based on this, and using a dynamic control field, a weighted joint cost equation is constructed by combining the observation equations of synchronous observations performed by a panoramic camera array, a 3D laser scanner, and a robotic arm system at known measurement points. This allows for the global optimal estimation of the platform pose of the track inspection platform. The weighted joint cost equation is expressed as follows:

[0163] (12)

[0164] in, Represent the weighted joint cost equation. The confidence weights of the panoramic camera array are represented. This indicates the confidence weight of the 3D laser scanner. This represents the confidence weight of the robotic arm system, and it is greater than the other two confidence weights. This represents the cost equation derived from the observation equation of the panoramic camera array. This represents the cost equation derived from the observation equation of the 3D laser scanner. The cost equation derived from the observation equations of the robotic arm system is represented by... Let represent the rotation matrix in the positioning and orientation determination of the track inspection platform to be solved. Let represent the translation vector in the positioning and attitude determination of the track inspection platform to be solved. Let represent the normalized direction vector from the camera to the measurement point, and subscript Indicates the serial number of the measuring point. Represents pixel coordinates, Let represent the intrinsic parameter matrix of the camera, and , Represents the platform coordinate system To a single camera coordinate system The rotation matrix, Indicates the camera serial number. This represents the world coordinates of a known measurement point, i.e., its three-dimensional coordinates in the global coordinate system. Represents the platform coordinate system To a single camera coordinate system The translation vector, This represents the three-dimensional coordinates of a known measurement point acquired by a 3D laser scanner. Represents the platform coordinate system To the coordinate system of the 3D laser scanner The rotation matrix, Represents the platform coordinate system To the coordinate system of the 3D laser scanner The translation vector, This represents the three-dimensional coordinates of known measurement points acquired in the base coordinate system of a multi-degree-of-freedom robotic arm. Let represent the rotation matrix from the platform coordinate system of the track inspection platform to the base coordinate system of the multi-degree-of-freedom robotic arm, and , Let represent the translation vector from the platform coordinate system of the track inspection platform to the base coordinate system of the multi-degree-of-freedom robotic arm, and , This indicates a normalization operation.

[0165] In this embodiment, in step S3, the process of solving the positioning and orientation of the track inspection platform in the global coordinate system using the least squares method is as follows:

[0166] Using the linear differential measurement model of the robotic arm system constructed above, the track inspection platform at time point is calculated. up to the time point The relative pose change is superimposed onto the system state of the previous moment (e.g., the six degrees of freedom of position and attitude, namely x-coordinate, y-coordinate, z-coordinate, yaw angle, roll angle, and pitch angle) to serve as the initial estimate of the platform pose of the current orbit inspection platform. , ),in, This represents the initial rotation matrix of the track inspection platform. This represents the initial translation vector of the track inspection platform.

[0167] In the initial estimate ( , Based on this, and using the weighted joint cost equation as the objective function, the LM iterative optimization algorithm is employed to optimize the pose parameters (i.e., the rotation matrix). Translation vector Nonlinear optimization is performed to solve the problem.

[0168] After the algorithm converges, the output is the positioning and orientation of the track inspection platform. In the weighted joint cost equation (i.e., equation (12)), the weights are allocated and the geometric constraint term of the robotic arm system is given the highest weight. Therefore, the accuracy of the positioning and orientation solution is significantly better than the traditional method that only relies on camera arrays and 3D laser scanners.

[0169] like Figure 3 As shown, according to one embodiment of the present invention, step S4, which involves generating a three-dimensional model of the tunnel by performing global three-dimensional holographic stitching based on the positioning pose using observation data from a panoramic camera array and a three-dimensional laser scanner, includes:

[0170] The observation data from the panoramic camera array and the 3D laser scanner are acquired and preprocessed separately. The observation data includes: the panoramic RGB images acquired by the panoramic camera array and the 3D point cloud of the 3D laser scanner.

[0171] The positioning and pose of the track inspection platform in the global coordinate system are acquired, and the observation data is mapped to the global coordinate system based on the positioning and pose;

[0172] The three-dimensional point cloud is spliced based on the positioning and pose solved based on the adjacent inspection positions to obtain a global three-dimensional point cloud skeleton;

[0173] Image point color information of the look-around RGB image corresponding to each point cloud point is extracted, and 3D texture mapping is performed on the three-dimensional point cloud skeleton to construct a three-dimensional model of the tunnel.

[0174] As shown in the figure, Figure 4 According to an embodiment of the present application, in step S5, in the step of complementary fusion of the observation data of the panoramic camera array and the three-dimensional laser scanner based on the three-dimensional model, bidirectional complementary fusion is performed based on the image points in the look-around RGB image collected by the panoramic camera array and the point cloud points in the three-dimensional point cloud collected by the three-dimensional laser scanner to obtain fused features, and the bidirectional complementary fusion includes:

[0175] The three-dimensional point cloud collected by the three-dimensional laser scanner is converted into discrete point cloud voxel features, and 2D image features are extracted based on the image points in the look-around RGB image; in this embodiment, VoxelNet is used as the encoder for the three-dimensional laser scanner to convert the three-dimensional point cloud into discrete point cloud voxel features (denoted as ); SwinTransformer is used as the encoder for the panoramic camera array to extract 2D image features (denoted as ) from the image points in the look-around RGB image.

[0176] The point cloud voxel features are projected to the 2D image features, the neighboring image points are searched and fused with weights to generate semantic enhanced point cloud voxel features; in this embodiment, the non-empty voxel center points of the generated point cloud voxel features are projected to the image plane of the 2D image features, K nearest neighbor image points at this position are searched, and a weighting fusion mechanism based on distance prior is constructed to obtain point cloud feature information with semantic perception, thereby generating semantic enhanced point cloud voxel features; thus, the calculation process is as follows:

[0177] ;

[0178] ;

[0179] wherein, represents the semantic enhanced point cloud voxel features, represents a commonly used function in deep learning, which is used to normalize the distance value and assign weights to the points based on the distance, a weighted vector composed of inverses of distances between image points and projection points of point cloud voxel features, a set of image semantic features closest to the projection point of the point cloud voxel feature, and , for each point cloud voxel, i.e. the number of neighborhood points, the number of feature channels, a semantic enhanced point cloud voxel feature, a nonlinear activation function for eliminating back features in learning, a linear transformation operation for dimension transformation and linear combination of input features, a point cloud voxel feature.

[0180] Based on this, the recognition ability of point cloud features for complex disease texture can be effectively improved.

[0181] The point cloud voxel feature is projected to generate a sparse depth map, and a dense depth feature map is obtained by using a depth completion method. After the dense depth feature map and the 2D image feature are spliced, dimension fusion is performed to generate a spatial enhanced image feature. In the embodiment, the calculation process of dimension fusion is as follows:

[0182] ;

[0183] wherein, the spatial enhanced image feature is used to supplement the collection position, surface relief, and depth perception ability of the image in the disease area, and avoid misrecognition and missed detection caused by factors such as occlusion, illumination, and parallax in the past image recognition, a 2D image feature, a dense depth feature map, a convolution layer operation, feature splicing.

[0184] The spatial enhanced image feature is promoted to the same three-dimensional space as the point cloud voxel feature, and a fusion feature is obtained after adaptive weighted fusion with the semantic enhanced point cloud voxel feature. In the embodiment, the fusion process is as follows:

[0185] ;

[0186] ;

[0187] wherein, a weight factor of fusion of two modalities on each voxel feature, a three-dimensional convolution operation, a Sigmoid activation function, Representing spatial augmented image features of three-dimensional voxelization, representing fusion features for representing three-dimensional disease features.

[0188] Based on this fusion mechanism, the fusion ratio of semantic and spatial information can be automatically adjusted according to the feature quality of the fusion area, and the robustness of subsequent recognition of various disease morphologies is enhanced.

[0189] According to an embodiment of the present application, in step S5, the fusion features are loaded into the target detection network for disease recognition, and in the step of obtaining disease attribute information, the fusion features are imported into the target detection based on the target detection network, and the fusion features are decoded to output the target frame of the disease and obtain the pixel coordinates of the disease on the two-dimensional expansion map, and can be expressed as , so as to facilitate subsequent back-projection conversion.

[0190] Combined with the camera intrinsic matrix K, the extrinsic matrix (rotation matrix and translation vector) of the image point corresponding to the disease, and the scale factor provided by the three-dimensional point cloud , the two-dimensional pixel coordinates are back-projected to the three-dimensional space to obtain the spatial position thereof in the unified global coordinate system, and the conversion form is as follows:

[0191] ;

[0192] ;

[0193] ;

[0194] wherein, represents the coordinate information of the measuring point (i.e. the first dynamic control point) in the camera coordinate system , represents the coordinate information of the measuring point (i.e. the first dynamic control point) in the platform coordinate system , represents the rotation matrix from the single camera coordinate system to the platform coordinate system , which is the transpose of the rotation matrix , represents the offset from the camera coordinate system C to the platform coordinate system , represents the three-dimensional space coordinates corresponding to the two-dimensional pixel coordinates , represents the rotation matrix from the world coordinate system to the platform coordinate system , which is the transpose of the rotation matrix The transpose of .

[0195] Integrating the above transformation forms, we can obtain the back projection calculation formula, which is expressed as:

[0196] ;

[0197] in, Two-dimensional pixel coordinates The corresponding three-dimensional spatial coordinates. For complex disease areas (such as cracked areas), multiple feature points can be selected for three-dimensional projection, and finally fitted into a three-dimensional line segment or curve to represent the spatial path of the crack.

[0198] After disease identification, a sequence of three-dimensional coordinate points of the diseased area is obtained by combining the three-dimensional spatial back projection results. The length of the disease is then calculated by accumulating Euclidean distances. At the same time, the width pixel value of the disease in the image is also considered. Target depth and distance With camera focal length Calculate the actual width of the disease in physical space. And it is represented as:

[0199] ;

[0200] ;

[0201] in, , Represents adjacent three-dimensional coordinate points within the disease. Indicates the index of the three-dimensional coordinate point. This indicates the number of three-dimensional coordinate points.

[0202] Furthermore, the centroid method is used to obtain the spatial principal positioning point of the lesion, which facilitates the subsequent robotic arm system to obtain the relevant physical properties of the detached lesion body through this spatial principal positioning point. The calculation formula for the spatial principal positioning point is as follows:

[0203] ;

[0204] in, This indicates the main spatial location of the disease, i.e., the centroid or center point obtained based on the centroid method.

[0205] Furthermore, the identified defects are recorded with fields such as defect number, center coordinates, length, width, identification confidence level, and acquisition time, forming a complete three-dimensional attribute information table for use by intelligent rail inspection vehicles and robotic arm systems. This table guides the robotic arm system in acquiring the physical properties of the defects and their detached parts. Thus, the identification of defects and the assignment of their three-dimensional coordinate information in three-dimensional holographic images are achieved. See [link to documentation].Figure 5 Therefore, the disease attribute information includes: disease number, center coordinates, length, width, recognition confidence, and collection time.

[0206] According to an embodiment of the present application, in step S6, based on the disease three-dimensional attribute information and the normal vector of the wall surface where the disease is located, the secondary inspection path of the rail inspection platform is planned, the rail inspection platform is guided to autonomously navigate to the disease area, and the working range of the mechanical arm system is obtained based on the disease three-dimensional attribute information and the normal vector of the wall surface where the disease is located. Therefore, the working range can be projected on the track plane of the intelligent rail inspection vehicle, and then the mileage information for the intelligent rail inspection vehicle to identify can be obtained, so that the intelligent rail inspection vehicle can be guided to navigate with high precision in the track plane, and the mechanical arm system can be controlled to perform wall adhesion detection on the disease in the direction of the wall surface normal vector after reaching the working position, to obtain the physical parameters such as the depth and volume of the disease void, and the corresponding disease point cloud. Therefore, step S6 includes:

[0207] Based on the disease three-dimensional attribute information, the disease center point coordinates and the corresponding wall surface normal vector are obtained; in this embodiment, in order to determine the wall adhesion detection direction of the mechanical arm system and the working area range of the intelligent rail inspection vehicle, the normal vector of the disease area point cloud needs to be extracted, and thus, based on the spatial master positioning point of all diseases , a point set is constructed. At the same time, all points in the point set are converted into a point set in a decentralized form.

[0208] Further, according to the point set , a covariance matrix is constructed, and the eigenvector corresponding to the minimum eigenvalue is taken as the normal vector . Therefore, the constructed covariance matrix is as follows:

[0209] ;

[0210] ;

[0211] Wherein, represents the total number of points in the point cloud participating in the calculation, , , , , , , , , respectively represent each component element in the point cloud covariance matrix, Represents the first covariance matrix after decomposition. 1 eigenvector Represents the first covariance matrix after decomposition. Each feature value.

[0212] Furthermore, based on the wall-hugging distance required by the robotic arm system's detection requirements... To obtain the location of the defect detected by the robotic arm system. Due to the point The point is suspended in the air, therefore this point... Further projection onto the track plane of the intelligent track inspection vehicle's movement Thus, it is possible to be in the orbital plane The corresponding navigation destination of the intelligent rail inspection vehicle is obtained. This allows for precise positioning of the intelligent rail inspection vehicle's running location; therefore, the projection process can be represented as:

[0213] ;

[0214] ;

[0215] in, Indicates belonging to the orbital plane The point on top.

[0216] Based on the integrated navigation system and through Algorithms and PID closed-loop control guide the track inspection platform to autonomously navigate to the defect area; in this embodiment, the three-dimensional holographic image trajectory point set from the first inspection... In the middle, with the orbital plane To achieve the goal, adopt The algorithm searches and generates a path from the current position. To the navigation destination Optimal path sequence An extended Kalman filter is introduced to adjust the state vector. Estimate the values ​​and use a dual-channel PID control law to adjust the linear velocity and angular velocity, as follows:

[0217] ;

[0218] ;

[0219] in, This indicates the linear velocity of the track inspection platform. Indicates the proportion of the linear velocity channel. This indicates the Euclidean distance deviation between the current position and the target point. This represents the integral of the linear velocity channel. The differential gain function representing the linear velocity channel. represents the angular velocity of the rail inspection platform, represents the proportion of the angular velocity channel, represents the angular error between the current yaw angle and the expected heading, represents the integral of the angular velocity channel, represents the differential gain function of the angular velocity channel, represents the time.

[0220] Thus, by controlling the intelligent rail inspection vehicle to travel along the rail plane until the intelligent rail inspection vehicle reaches the position , and based on the ideal position of the disease area projected to the tunnel rail plane , the stopping condition of the intelligent rail inspection vehicle is obtained, which can be specifically represented as:

[0221] ;

[0222] wherein, represents the preset position error tolerance threshold of the intelligent rail inspection vehicle reaching the target position, represents the preset heading angle error tolerance threshold of the intelligent rail inspection vehicle reaching the target attitude.

[0223] When the rail inspection platform reaches the working position, the mechanical arm system performs smooth wall adhesion operation on the disease based on inverse kinematics and spatial interpolation trajectory planning, and obtains the disease physical parameters and disease point cloud; in the embodiment, the target pose matrix of the multi-degree-of-freedom mechanical arm end of the mechanical arm system is constructed , the forward kinematics is established based on the D-H model , and is represented as:

[0224] ;

[0225] wherein, represents the rotation matrix constructed with the normal vector as the target attitude, represents the concatenated product of the homogeneous transformation matrices between the coordinate systems of each link of the multi-degree-of-freedom mechanical arm, represents the vector representation of each joint angle of the multi-degree-of-freedom mechanical arm.

[0226] The optimization function is constructed, and is represented as:

[0227] ;

[0228] wherein, represents the optimization function.

[0229] The damping least squares method is used for numerical iteration, and is represented as:

[0230] ;

[0231] ;

[0232] in, Indicates the first Vector representation of the joint angles of a multi-degree-of-freedom robotic arm in each iteration. Indicates the first Vector representation of the joint angles of a multi-degree-of-freedom robotic arm in each iteration. Let represent the Jacobian matrix, used to describe the mapping relationship between joint angle changes and end-effector attitude changes, which can be simplified to , The pseudo-inverse of the Jacobian matrix can be simplified to: , Indicates the damping factor. Represents the identity matrix.

[0233] Furthermore, Cartesian space interpolation is used to generate smooth... The details are as follows:

[0234] Position interpolation: ;

[0235] Attitude interpolation: ;

[0236] in, Indicates the uniform time parameter. Indicates time The interpolation value at time is the fulcrum. This represents the spherical linear interpolation method. This represents the initial position vector of the robotic arm's end effector at the start of the interpolation. This represents the desired target position vector of the robotic arm's end effector at the end of the interpolation. This indicates the attitude at the interpolation moment.

[0237] The pose of the multi-DOF robotic arm in each frame after interpolation is as follows: ;

[0238] posture The corresponding joint angle sequence is obtained by solving inverse kinematics. The servo system of a multi-degree-of-freedom robotic arm accepts joint angle sequences. After control is achieved, a smooth wall-hugging operation is performed to achieve precise docking between the robotic arm and the target disease.

[0239] According to one embodiment of the present invention, in step S7, the point cloud of the disease and the three-dimensional model are registered using ICP, and the disease is corrected based on the registration result. In this step, the point cloud of the disease and the three-dimensional model are registered using a registration optimization function, which is expressed as follows:

[0240] ;

[0241] wherein, represents the best rotation transformation matrix between the disease local point cloud to be solved and the three-dimensional model point cloud, represents the best translation transformation vector between the disease local point cloud to be solved and the three-dimensional model point cloud, represents the disease point cloud collected by the smooth wall pasting operation, represents the disease point cloud in the three-dimensional model, represents the index of the corresponding point pair in the point cloud;

[0242] The disease center coordinate after disease correction is:

[0243] ;

[0244] wherein, represents the corrected disease center coordinate, represents the disease center coordinate in the three-dimensional model, i.e. the disease center coordinate before correction.

[0245] According to an embodiment of the present application, a tunnel disease global positioning system fusing a high-precision constraint mechanical arm, comprising:

[0246] The track inspection platform comprises an intelligent track inspection vehicle, a panoramic camera array, a three-dimensional laser scanner, a mechanical arm system and a combined navigation system mounted on the intelligent track inspection vehicle;

[0247] The global coordinate system construction module is used to set the global coordinate system of the tunnel;

[0248] The dynamic control field construction module is used to establish a joint adjustment model of the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system for the rephotographing process of the same measuring point at adjacent time points, and to construct a dynamic control field of the known measuring point;

[0249] The positioning and orientation solving module is based on the dynamic control field, and uses the panoramic camera array, the three-dimensional laser scanner and the mechanical arm system to perform synchronous observation on the known measuring point, constructs a weighted joint cost equation, and solves the positioning and orientation of the track inspection platform in the global coordinate system through the least square method;

[0250] The holographic splicing module is based on the positioning and orientation, and uses the observation data of the panoramic camera array and the three-dimensional laser scanner to perform global three-dimensional holographic splicing to generate a three-dimensional model of the tunnel;

[0251] The disease primary identification module fuses observation data of the panoramic camera array and the three-dimensional laser scanner based on the three-dimensional model, and loads fused features into a target detection network for disease identification to obtain disease three-dimensional attribute information.

[0252] The disease secondary identification module plans a secondary inspection path of the rail inspection platform based on disease three-dimensional attribute information and a normal vector of a wall surface where the disease is located, guides the rail inspection platform to perform disease inspection, and obtains disease physical parameters and a disease point cloud.

[0253] The disease correction module performs ICP registration of the disease point cloud and the three-dimensional model, performs disease correction according to a registration result, and outputs a disease positioning result.

[0254] In the embodiment, in each wall adhesion action of the mechanical arm system, the multi-degree-of-freedom mechanical arm obtains three or more wall adhesion control points according to a known pose and a ranging result, three-dimensional coordinates of the control points are calculated by a sensor origin, a transmission direction, and a measured distance, and the control points are distributed on a local disease or a structure region.

[0255] Specific limitations of the tunnel disease global positioning system fusing the high-precision constraint of the mechanical arm can be seen in the limitations of the tunnel disease global positioning method fusing the high-precision constraint of the mechanical arm, which will not be repeated here. Each module in the above-mentioned tunnel disease global positioning system fusing the high-precision constraint of the mechanical arm can be realized by software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0256] The above is only an example of a specific solution of the present application, and for devices and structures not described in detail, it should be understood that general devices and general methods in the art are used to implement them.

[0257] The above only describes one solution of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A tunnel disease global positioning method fusing high-precision constraints of a mechanical arm, characterized in that, The method comprises the following steps: S1. Constructing a rail inspection platform, wherein the rail inspection platform comprises: an intelligent rail inspection vehicle, a panoramic camera array, a three-dimensional laser scanner, a mechanical arm system, and a combined navigation system mounted on the intelligent rail inspection vehicle; S2. Setting a global coordinate system of the tunnel, establishing a joint adjustment model of the panoramic camera array, the three-dimensional laser scanner, and the mechanical arm system based on a rephotographing process of the same measuring point at adjacent time points, and constructing a dynamic control field of the known measuring point; S3. Based on the dynamic control field, performing synchronous observation on the known measuring point using the panoramic camera array, the three-dimensional laser scanner, and the mechanical arm system, constructing a weighted joint cost equation, and solving the positioning and pose of the rail inspection platform in the global coordinate system by the least square method; S4. Based on the positioning and pose, performing global three-dimensional holographic splicing using the observation data of the panoramic camera array and the three-dimensional laser scanner to generate a three-dimensional model of the tunnel; S5. Based on the three-dimensional model, complementarily fusing the observation data of the panoramic camera array and the three-dimensional laser scanner, and loading the fused features into a target detection network for disease identification to obtain three-dimensional attribute information of the disease and complete a first inspection; S6. Based on the three-dimensional attribute information of the disease and the normal vector of the wall surface where the disease is located, planning a second inspection path of the rail inspection platform, guiding the rail inspection platform to perform disease inspection, and obtaining physical parameters and disease point clouds of the disease; S7. ICP registration of the disease point clouds and the three-dimensional model, disease correction according to the registration result, and completion of the second inspection.

2. The tunnel disease global positioning method of fusing the high-precision constraint of the mechanical arm, according to claim 1, characterized in that, In step S1, in the step of constructing the rail inspection platform, the panoramic camera array comprises: a surround-view camera array and a reference transfer camera array; The surround-view camera array is used to obtain surround-view images at each inspection position in the tunnel, and has at least two surround-view cameras; The reference transfer camera array is used to transfer control information of the global coordinate system at both ends of the tunnel to the inspection area in the tunnel, and has a forward-looking camera and a rear-view camera; The three-dimensional laser scanner is used to obtain three-dimensional point clouds at each inspection position in the tunnel; The mechanical arm system comprises: a multi-degree-of-freedom mechanical arm and a range finder mounted at the end of the multi-degree-of-freedom mechanical arm; The multi-degree-of-freedom mechanical arm is used to control the pose and position of the range finder, and the range finder is used to measure the measuring point in the tunnel by wall attachment; The combined navigation system is used for navigation of the intelligent rail inspection vehicle in the tunnel, and has at least one of an IMU device, a GNSS device, and a DMI device.

3. The tunnel disease global positioning method of fusing the high-precision constraint of the mechanical arm, according to claim 2, characterized in that, In step S2, in the step of establishing a joint adjustment model of the panoramic camera array, the three-dimensional laser scanner, and the mechanical arm system based on a rephotographing process of the same measuring point at adjacent time points to construct a dynamic control field of the known measuring point, comprising: S21. Based on the rephotographing process of the same measuring point at adjacent time points, linear difference measurement models of the panoramic camera array, the three-dimensional laser scanner, and the mechanical arm system are respectively established; S22. Based on the observation constraints in the dynamic network formed by the track inspection platform movement, the panoramic camera array, the three-dimensional laser scanner, and the mechanical arm system, and under the highest confidence weight given to the geometric constraints of the mechanical arm system, the three linear difference measurement models are constructed into a unified weighted least squares adjustment system to solve the relative pose parameters of the track inspection platform and the displacement of the measuring points, and to construct the dynamic control field.

4. The tunnel disease global positioning method of fusing the high-precision constraint of the mechanical arm, according to claim 3, characterized in that, In step S21, in the process of establishing the linear difference measurement models of the panoramic camera array, the three-dimensional laser scanner, and the mechanical arm system based on the rephotographing of the same measuring point at adjacent time points, the linear difference measurement model of the mechanical arm system is represented as: wherein, denotes a linear difference measurement model of the robot system at adjacent time points , , denotes a transformation matrix from the platform coordinate system of the rail inspection platform to the base coordinate system of the multi-degree-of-freedom robot in the robot system , denotes the sum of the displacement of the intelligent rail inspection vehicle and the three-dimensional displacement of the measurement point itself, denotes the cumulative basis function coefficient of the B-spline curve at the corresponding time, denotes the Lie algebra control point basis matrix in the SE(3) space, denotes the three-dimensional coordinates of the measurement point in the platform coordinate system at the time point , denotes the displacement of the measurement point in the platform coordinate system at the time point , denotes the summation index of the B-spline basis function; The linear difference measurement model of the panoramic camera array is represented as: wherein, denotes a linear difference measurement model of the panoramic camera array at adjacent time points , , denotes a spatial depth factor of the measurement point at time point with respect to the camera optical center, denotes the camera intrinsic parameters, denotes the transformation matrix from the platform coordinate system to the camera coordinate system , denotes a zero vector for matrix dimension matching; The linear difference measurement model of the three-dimensional laser scanner is represented as: wherein denotes a linear difference measurement model of the three-dimensional laser scanner at adjacent time points , , denotes a transformation matrix from the platform coordinate system to the three-dimensional laser scanner coordinate system .

5. The tunnel disease global positioning method of fusing the high-precision constraint of the mechanical arm, according to claim 4, characterized in that, In step S22, the observation constraints in the dynamic network include: the same-position multi-module pose consistency constraint and the cross-position same-measuring-point displacement consistency constraint; The same-position multi-module pose consistency constraint is that, under the same inspection position, different measuring points observed by the panoramic camera array, the three-dimensional laser scanner, and the mechanical arm system correspond to the same relative pose parameters; The cross-position same-measuring-point displacement consistency constraint is that, under different inspection positions, the same measuring point observed by the panoramic camera array, the three-dimensional laser scanner, and the mechanical arm system corresponds to the same displacement parameters.

6. The tunnel disease global positioning method of fusing the high-precision constraint of the mechanical arm, according to claim 5, characterized in that, In step S3, based on the dynamic control field, the synchronous observation of the panoramic camera array, the three-dimensional laser scanner, and the mechanical arm system is performed on the known measuring points, and the weighted joint cost equation is constructed, which is represented as: wherein, represents a weighted joint cost equation, represents a confidence weight of the panoramic camera array, represents a confidence weight of the three-dimensional laser scanner, represents a confidence weight of the robotic arm system, and is greater than the other two confidence weights, represents a cost equation derived from an observation equation of the panoramic camera array, represents a cost equation derived from an observation equation of the three-dimensional laser scanner, represents a cost equation derived from an observation equation of the robotic arm system, represents a rotation matrix in the positioning and orientation of the track inspection platform to be solved, represents a translation vector in the positioning and orientation of the track inspection platform to be solved, represents a normalized direction vector from the camera to the measuring point, and , subscript represents the serial number of the measuring point, represents the pixel coordinates of the measuring point, represents the intrinsic matrix of the camera, and , represents a rotation matrix of the platform coordinate system to the single camera coordinate system , represents the camera serial number, represents the world coordinates of the known measuring point, i.e. the three-dimensional coordinates in the global coordinate system, represents a translation vector of the platform coordinate system to the single camera coordinate system , represents the three-dimensional coordinates of the known measuring point collected by the three-dimensional laser scanner, represents a rotation matrix of the platform coordinate system to the three-dimensional laser scanner coordinate system , represents a translation vector of the platform coordinate system to the three-dimensional laser scanner coordinate system , represents the three-dimensional coordinates of the known measuring point collected by the multi-degree-of-freedom robotic arm in the base coordinate system of the multi-degree-of-freedom robotic arm, represents a rotation matrix of the platform coordinate system of the track inspection platform to the base coordinate system of the multi-degree-of-freedom robotic arm, and , represents a translation vector of the platform coordinate system of the track inspection platform to the base coordinate system of the multi-degree-of-freedom robotic arm, and , represents a normalization operation.

7. The tunnel disease global positioning method of fusing the high-precision constraint of the mechanical arm, according to claim 6, characterized in that, In step S4, based on the positioning pose, the global three-dimensional holographic splicing is performed on the observation data of the panoramic camera array and the three-dimensional laser scanner to generate the three-dimensional model of the tunnel, which includes: The observation data of the panoramic camera array and the three-dimensional laser scanner are acquired and preprocessed respectively; wherein, the observation data includes: the ring-view RGB image collected by the panoramic camera array and the three-dimensional point cloud of the three-dimensional laser scanner; The positioning and pose of the track inspection platform in the global coordinate system are acquired, and the observation data is aligned to the global coordinate system based on the positioning and pose; The three-dimensional point cloud is spliced based on the positioning and pose solved at the adjacent inspection positions to obtain the global three-dimensional point cloud skeleton; The image point color information of the ring-view RGB image corresponding to each point cloud point is extracted, and the three-dimensional point cloud skeleton is subjected to 3D texture mapping to construct the three-dimensional model of the tunnel.

8. The tunnel disease global positioning method of fusing the high-precision constraint of the mechanical arm, according to claim 7, characterized in that, In step S5, in the process of complementary fusion of the observation data of the panoramic camera array and the three-dimensional laser scanner based on the three-dimensional model, bidirectional complementary fusion is performed on the image points in the ring-view RGB image collected by the panoramic camera array and the point cloud points in the three-dimensional point cloud collected by the three-dimensional laser scanner to obtain the fused features, and it includes: The three-dimensional point cloud collected by the three-dimensional laser scanner is converted into discrete point cloud voxel features, and 2D image features are extracted based on the image points in the ring-view RGB image; The point cloud voxel features are projected to the 2D image features, neighboring image points are retrieved and fused with weights to generate semantic enhanced point cloud voxel features; The point cloud voxel feature is projected to generate a sparse depth map, and a depth completion method is used to obtain a dense depth feature map, the dense depth feature map is spliced with a 2D image feature to perform dimension fusion, and a spatial enhanced image feature is generated; The spatial enhanced image feature is lifted to the same three-dimensional space as the point cloud voxel feature, and the spatial enhanced image feature is adaptively weighted and fused with a semantic enhanced point cloud voxel feature to obtain the fusion feature; In step S5, the fusion feature is loaded into a target detection network for disease identification, and disease attribute information is obtained, wherein the disease attribute information includes: disease number, center coordinates, length, width, recognition confidence, and collection time.

9. The tunnel disease global positioning method of fusing the high-precision constraint of the mechanical arm, according to claim 8, characterized in that, In step S6, based on the disease three-dimensional attribute information and the normal vector of the wall surface where the disease is located, a secondary inspection path of the rail inspection platform is planned, the rail inspection platform is guided to autonomously navigate to the disease area, and disease physical parameters and disease point clouds are obtained, including: Based on the disease three-dimensional attribute information, the disease center point coordinates and the corresponding wall surface normal vector are obtained; Based on the combination navigation system and through The algorithm and the PID closed loop control guide the track inspection platform to autonomously navigate to the disease area. After the rail inspection platform reaches the working position, the mechanical arm system performs smooth wall pasting operation on the disease based on inverse kinematics and spatial interpolation trajectory planning, and disease physical parameters and disease point clouds are obtained; In step S7, the disease point cloud is ICP registered with the three-dimensional model, and disease correction is performed according to the registration result, wherein the disease point cloud and the three-dimensional model are based on a registration optimization function, and the registration optimization function is represented as: wherein, represents the best rotation transformation matrix between the disease local point cloud to be solved and the three-dimensional model point cloud, represents the best translation transformation vector between the disease local point cloud to be solved and the three-dimensional model point cloud, represents the disease point cloud collected by the smooth wall pasting operation, represents the disease point cloud in the three-dimensional model, represents the index of the corresponding point pair in the point cloud; The disease center coordinates after disease correction are: wherein, represents the corrected disease center coordinate, represents the disease center coordinate in the three-dimensional model, i.e. the uncorrected disease center coordinate.

10. A tunnel disease global positioning system fusing high-precision constraints of a mechanical arm, characterized in that, including: The rail inspection platform includes: an intelligent rail inspection vehicle, a panoramic camera array, a three-dimensional laser scanner, a mechanical arm system, and a combined navigation system mounted on the intelligent rail inspection vehicle; A global coordinate system construction module is configured to set a global coordinate system of the tunnel; A dynamic control field construction module is configured to establish a joint adjustment model of the panoramic camera array, the three-dimensional laser scanner, and the mechanical arm system for the rephotographing process of the same measurement point at adjacent time points, and to construct a dynamic control field of the known measurement point; A positioning and orientation solving module is configured to perform synchronous observation on the known measurement point by using the panoramic camera array, the three-dimensional laser scanner, and the mechanical arm system based on the dynamic control field, to construct a weighted joint cost equation, and to solve the positioning and orientation of the rail inspection platform in the global coordinate system by using the least square method; A holographic splicing module is configured to perform global three-dimensional holographic splicing based on the positioning and orientation by using the observation data of the panoramic camera array and the three-dimensional laser scanner, and to generate a three-dimensional model of the tunnel; A disease primary identification module is configured to fuse the observation data of the panoramic camera array and the three-dimensional laser scanner based on the three-dimensional model, to load the fusion feature into a target detection network for disease identification, and to obtain disease three-dimensional attribute information; A disease secondary identification module is configured to plan a secondary inspection path of the rail inspection platform based on the disease three-dimensional attribute information and the normal vector of the wall surface where the disease is located, to guide the rail inspection platform to perform disease inspection, and to obtain disease physical parameters and disease point clouds; A disease correction module is configured to ICP register the disease point cloud with the three-dimensional model, to perform disease correction according to the registration result, and to output disease positioning results.

Images