Tunnel leakage global positioning method and system of contour enhanced segmentation network based on panorama-infrared double cameras

Abstract

The invention provides a tunnel leakage global positioning method and system based on a contour enhanced segmentation network of panorama-infrared double cameras, and the method comprises the steps: capturing a 360-degree panorama image of a tunnel through a double-camera system integrating a double-fisheye camera and a thermal infrared camera, converting the panorama image into a plane image through an equidistant cylinder mapping method, and carrying out the global positioning of the tunnel leakage. The invention provides a tunnel wall segmentation method from a cylinder to a plane image to realize plane display of a tunnel wall surface, and meanwhile, the invention constructs an infrared image leakage segmentation network based on CEIL-Net, the network combines a contour enhancement technology and an attention mechanism, and the precision and robustness of leakage area segmentation are significantly improved. Besides, the leakage area in the infrared image is accurately mapped to the plane image of the tunnel through homography matrix calculation based on the camera attitude, and comprehensive visualization of the leakage position is achieved. The tunnel leakage area can be effectively identified and positioned, and the method is suitable for tunnel health monitoring and maintenance.

Description

Technical Field

[0001] The present invention belongs to the field of tunnel health monitoring and maintenance, and in particular relates to a tunnel leakage global positioning method and system based on a contour enhancement segmentation network of a panoramic-infrared dual camera. Background Art

[0002] With the rapid development of modern transportation networks, tunnels, as key infrastructure connecting mountainous areas and cities, are receiving increasing attention for their safe operation. Tunnel leakage, that is, the occurrence of water infiltration in the tunnel structure, will not only weaken the stability of the tunnel structure and increase maintenance costs, but also, under the influence of freeze-thaw cycles, is more likely to cause structural rupture and even serious safety accidents. In addition, leakage will also cause the internal environment humidity to increase, providing conditions for microbial growth, accelerating the corrosion of equipment inside the tunnel and shortening its service life. Therefore, timely detection and location of tunnel leakage areas are crucial to ensuring driving safety and extending the service life of tunnels. However, traditional manual detection methods have many disadvantages. First, the manual detection process is time-consuming and labor-intensive, highly subjective, and difficult to quantify, resulting in low detection efficiency and unable to meet the safety needs of modern tunnels. Secondly, the internal environment of the tunnel is complex, with influencing factors such as strong light and shadows. It is difficult for manual detection to accurately determine the location and extent of water leakage, and it is easy to miss or misjudge. In addition, the complex morphology and huge scale of the tunnel also bring great challenges to manual detection. Therefore, it is necessary to develop an efficient and real-time monitoring system that can automatically monitor tunnel leakage continuously and provide timely warnings.

[0003] In recent years, the rapid development of deep learning technology has not only had a profound impact in the field of computer science, but also has spawned disruptive solutions in many fields such as biology, computational mathematics, materials science, and industrial engineering. At the same time, deep learning algorithms have also begun to emerge in the field of civil engineering, providing new ideas and methods for solving many difficult problems related to tunnel disease detection.

[0004] Although the current tunnel disease detection technology has made some progress, it is still insufficient in terms of global display of the disease location. Due to the narrow, tortuous and complex tunnel space, ordinary cameras are difficult to capture continuous tunnel images due to limited field of view. Therefore, most detection methods focus on local detection of the tunnel and cannot fully display the entire tunnel structure, making it difficult to form a global understanding of the tunnel's health status.

[0005] In previous studies, tunnel leakage detection methods mostly rely on visible light cameras, but in dimly lit environments, visible light cameras have difficulty capturing enough details, resulting in reduced detection accuracy. In contrast, infrared cameras show unique advantages in tunnel leakage detection due to their high sensitivity to temperature changes. However, since infrared imaging generally cannot provide resolution and edge clarity comparable to visible light cameras, the outline of the leakage area may appear blurred during image segmentation. Especially in the scenario of tunnel water leakage, the thermal difference between wet and dry areas may not be significant enough, making it more difficult to identify small leakage points through infrared images.

[0006] In the study of tunnel leakage detection, existing technologies mainly focus on visible light images with rich texture features. However, if these methods are directly applied to the infrared image projection of tunnel leakage, they will face a series of challenges. The specific environmental conditions inside the tunnel, such as dim light, non-uniform light distribution, and interference from moisture, make the correspondence between infrared images and panoramic images more complicated. In addition, infrared images usually show blurred outlines and missing details in the captured leakage areas, which further increases the difficulty of accurately projecting these images into the panoramic view. Summary of the invention

[0007] In view of the problems existing in the prior art, the present invention provides a method and system for global positioning of tunnel leakage based on a contour enhancement segmentation network of a panoramic-infrared dual camera, which overcomes the viewing angle limitation and complexity of image data management of traditional single-camera systems in the narrow space inside the tunnel. It also has significant technical advantages and practical application value in tunnel leakage detection and safety monitoring, and can provide more reliable and accurate technical support for tunnel health monitoring. In order to solve the above technical problems, the present invention provides the following technical solutions: A method for global positioning of tunnel leakage based on a contour enhancement segmentation network of a panoramic-infrared dual camera, comprising the following steps:

[0008] S1. Collect 360-degree panoramic images of the tunnel through dual fisheye cameras and collect thermal images of the leakage area through thermal infrared cameras;

[0009] S2, using the equidistant cylindrical mapping method to process the 360-degree panoramic image of the tunnel collected by the dual fisheye cameras, and converting the highly distorted spherical image into a planar panoramic image;

[0010] S3, separating the tunnel wall from the non-wall elements in the planar panoramic image, and expanding the tunnel wall into a rectangular image;

[0011] S4. Construct an infrared image leakage segmentation network based on CEIL-Net, take the thermal imaging of the leakage area as input, and the edge recognition result of the leakage area as output to train the infrared image leakage segmentation network and obtain the leakage area segmentation model;

[0012] S5. Input the thermal imaging of the leakage area into the leakage area segmentation model to obtain the edge recognition result of the leakage area; then use the panoramic mapping method of the leakage area based on the camera posture to map the edge recognition result of the leakage area to the plane image of the tunnel to achieve accurate positioning of the leakage position.

[0013] Furthermore, the isotropic cylindrical mapping method in the aforementioned step S2 includes the following sub-steps:

[0014] S2.1. Establish a three-dimensional rectangular coordinate system with the optical center of the lens as the origin, where the XYZ axes satisfy the right-hand rule, with the X axis coinciding with the optical axis of the lens and pointing to the outside of the camera; the Z axis points to the top of the lens; and the Y axis points to the right side of the lens;

[0015] S2.2. For any point P in space, the longitude of point P1 projected onto the unit sphere along the straight line OP is expressed as: The angle between the straight line projected onto the XOY plane and the X-axis is: The left side of the X axis satisfy: On the right satisfy: The latitude of point P is expressed as the angle θ between the straight line OP and the Z axis, where θ above the XOY plane satisfies: 0≤θ≤π / 2, and θ below the XOY plane satisfies: -π / 2≤θ≤0;

[0016] S2.3. After the spherical image is expanded, the horizontal field of view is 2π, and the vertical field of view is π, that is, the aspect ratio of the panoramic image is 2:1. The ordinate of the image is converted into latitude, and the abscissa is converted into longitude. The longitude and latitude of the dual fisheye images are mapped to a common plane. Let the width of the panoramic image be W1 and the height be H1. The longitude and latitude coordinates of any point (i, j) in the image are obtained by the following conversion formula:

[0017]

[0018] Where i and j represent the pixel position of the point in the horizontal and vertical directions respectively.

[0019] Furthermore, the aforementioned step S3 includes the following sub-steps:

[0020] S3.1. Establish a three-dimensional Cartesian world coordinate system at the center of the tunnel, with the origin at the center of the tunnel bottom, the X-axis along the tunnel, the Y-axis perpendicular to the tunnel bottom, and the Z-axis vertically upward; the radius is R, the length of the tunnel section to be unfolded is 2R, and the unfolded length of the tunnel wall is the circumference of the semi-cylinder πR, so the unfolded view of the tunnel wall forms a rectangle with a size of πR×2R, and then generate a canvas with an aspect ratio of W2:H2=2×π to simulate the image of the tunnel wall after unfolding;

[0021] S3.2. Traverse each pixel point on the canvas. For each pixel point (μ, ν) on the canvas, calculate the corresponding world coordinate P (x, y, z) on the tunnel wall according to its pixel coordinates. The conversion relationship is as follows:

[0022]

[0023] S3.3. Establish a spherical coordinate system at the origin of the world coordinate system to simulate the shooting position of the dual fisheye panoramic camera in the center of the tunnel, and then convert the world coordinates into spherical coordinates

[0024]

[0025] S3.4. Use the equidistant cylindrical mapping principle to determine the corresponding position (i, j) of the pixel point (μ, ν) on the canvas in the panoramic image, assign the color value of the corresponding pixel in the panoramic image to the pixel on the canvas, and realize the expansion of the image of the tunnel wall into a plane image.

[0026] Furthermore, the aforementioned step S4 for image leakage segmentation based on the CEIL-Net network includes the following sub-steps: S4.1, introducing the contour filter kernel OLK in the first two cross-layer connections of the CEIL-Net network, significantly enhancing the edge features in the image in the shallow convolution layer, adding the SE module after every two convolution operations of the CEIL-Net; and combining the cross entropy loss, Dice loss and edge-aware loss into a total combined loss function to improve the model performance;

[0027] S4.2, the contour filter kernel OLK takes the weight of the central pixel as positive and the weight of the surrounding pixels as negative, as shown in the following formula:

[0028]

[0029] S4.3. After every two convolution operations in the model, a (SE) module is added to further extract the channel information of the feature map.

[0030] S4.4, the total combined loss function is as follows:

[0031] L total =α·LCE +β·L Dice +γ·L Edge

[0032] Among them, L CE is the cross entropy loss, L Dice is the Dice loss, L Edge is the edge-aware loss, α, β, γ are weight coefficients used to adjust the contribution of different loss functions to the total loss;

[0033] In the formula,

[0034]

[0035] Where N is the total number of pixels, y i is the true label of the i-th pixel, p i is the probability that the pixel is predicted as positive, i.e. leakage, and Represent the gradients of the true label and the predicted label at the i-th pixel respectively.

[0036] Furthermore, in the aforementioned step S5, the panoramic mapping method of the leakage area based on the camera posture includes the following steps:

[0037] S5.1, after performing stereo segmentation on the panoramic unfolded image, projection images in three directions of left, top and right are obtained;

[0038] S5.2, a thermal infrared calibration plate was used, the calibration plate was heated to a preset temperature, and a preset number of images were captured using two-mode cameras at the same time, and then calibrated using the Zhang Zhengyou calibration method to obtain the internal parameters of each camera, including: K vis For panoramic cameras, K ir For thermal infrared cameras; external parameters include: R vis 、T vis For panoramic cameras, and R ir 、T ir For thermal infrared cameras;

[0039] S5.3. Solve the homography matrix between the two cameras, let the world coordinate system W and the camera coordinate system x1y1z1 of the thermal infrared camera coincide, and calculate the relative position of the panoramic camera relative to the thermal infrared camera based on the calibrated internal and external parameters of the two cameras as [R relative T relative ], and calculate the homography matrix H according to the above camera posture.

[0040]

[0041] S5.4. The leakage area in the panoramic segmentation image is projected onto the tunnel wall expansion image. The conversion relationship between the point coordinates (x, y, z) of the leakage area in the panoramic segmentation image in the O-XYZ world coordinate system and the pixel coordinates (μ, ν) of the tunnel wall expansion image is:

[0042]

[0043] Where ψ is the angle between the projection line of the point on the segmented image and the line connecting point O in the O-YZ plane and the Z axis.

[0044] The present invention also provides a tunnel leakage global positioning system based on a contour enhancement segmentation network of a panoramic-infrared dual camera, comprising: a camera system, a camera fixing and adjustment mechanism, a mobile support system, a power management module, a data transmission module, and a data processing module; the camera system comprises a dual fisheye camera and an infrared camera;

[0045] The dual fisheye camera uses two 180-degree fisheye lenses to shoot at the left and right sides of the tunnel respectively, so as to capture a 360-degree panoramic image; the focal length of each fisheye camera is 7.2 mm, and a 1 / 2-inch CMOS photosensitive element is used;

[0046] The infrared camera uses an uncooled focal plane thermal infrared detector with a focal length of 3.2 mm, a field of view of 55.6°×42°, and a resolution of 256×192, which is used to capture thermal images inside the tunnel;

[0047] The camera fixing and adjusting mechanism is used to install the dual fisheye camera and the infrared camera on the multifunctional mobile trolley. The camera is placed on the top layer, and a mobile power supply and a computer are configured on the bottom layer. The computer is used to control the camera's shooting screen in real time, save the collected picture data and process it;

[0048] The power management module is used to provide continuous power to the camera system and the computer;

[0049] The data transmission module transmits the data collected by the camera system to the computer;

[0050] The data processing module is built into the computer and includes a plane panorama conversion unit, a rectangular image generation unit, a leakage area segmentation model training unit, and a leakage position positioning unit;

[0051] The panoramic image conversion unit is used to process the 360-degree panoramic image of the tunnel collected by the dual fisheye cameras using an equidistant cylindrical mapping method, and convert the highly distorted spherical image into a flat panoramic image;

[0052] A rectangular image generating unit, used for separating the tunnel wall from non-wall elements in the planar panoramic image, and expanding the tunnel wall into a rectangular image;

[0053] The leakage area segmentation model training unit is used to build an infrared image leakage segmentation network based on CEIL-Net, and uses the leakage area thermal imaging as input and the edge recognition result of the leakage area as output to train the infrared image leakage segmentation network and obtain the leakage area segmentation model;

[0054] The leakage position positioning unit is used to input the thermal imaging of the leakage area into the leakage area segmentation model to obtain the edge recognition result of the leakage area; then, the edge recognition result of the leakage area is mapped onto the plane image of the tunnel using the panoramic mapping method of the leakage area based on the camera posture to achieve accurate positioning of the leakage position.

[0055] Compared with the prior art, the beneficial technical effects of the present invention using the above technical solution are as follows:

[0056] (1) Panoramic coverage and precise positioning: By using a dual fisheye camera system, the present invention can cover a 360-degree panoramic view of the tunnel interior, effectively overcoming the limitations of traditional camera view. This panoramic coverage ensures comprehensive detection of the tunnel leakage area without blind spots. At the same time, the projection conversion technology from cylindrical to flat images can accurately display the tunnel wall surface and provide precise positioning of tunnel leakage.

[0057] (2) High-precision leakage area segmentation: The CEIL-Net model used combines contour enhancement technology and attention mechanism to significantly improve the model's ability to identify the edges of leakage areas, thereby improving the limitations of traditional models in infrared image leakage segmentation tasks. This technology significantly improves the accuracy and robustness of leakage area segmentation, especially when dealing with blurred edge areas.

[0058] (3) Applicable to low-light environments: By utilizing the capabilities of infrared thermal imaging cameras, the present invention is not affected by ambient light and can effectively perform leakage detection even in dimly lit tunnels, thereby enhancing the applicability and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is the overall flow chart of the invention.

[0060] Figure 2 Figure 2 is a diagram of the equidistant cylindrical mapping method. (a) is the principle diagram of the equidistant cylindrical mapping, and (b) is the panoramic expansion diagram of the dual fisheye image.

[0061] Figure 3 It is the unfolded diagram of the tunnel wall; (a) is the schematic diagram of the unfolded tunnel wall, (b) is the unfolded rectangular diagram; (c) is the panoramic diagram of the tunnel wall.

[0062] Figure 4 This is the structure diagram of the CEIL-Net model.

[0063] Figure 5 Schematic diagram of SE module.

[0064] Figure 6 This is the joint calibration diagram of the dual-camera system.

[0065] Figure 7 Schematic diagram of dual-camera joint calibration in five directions.

[0066] Figure 8 This is a diagram showing the arrangement of measurement points for data collection.

[0067] Fig. 9 This is the training curve diagram of the CEIL-Net network.

[0068] Fig.10 Tunnel leakage map. DETAILED DESCRIPTION

[0069] In order to better understand the technical content of the present invention, specific embodiments are given and described as follows in conjunction with the accompanying drawings.

[0070] Various aspects of the invention are described herein with reference to the accompanying drawings, in which many illustrative embodiments are shown. The embodiments of the invention are not limited to those described in the accompanying drawings. It should be understood that the invention is implemented by any of the various concepts and embodiments described above, as well as the concepts and embodiments described in detail below, because the concepts and embodiments disclosed in the invention are not limited to any implementation. In addition, some aspects disclosed in the invention may be used alone or in any appropriate combination with other aspects disclosed in the invention.

[0071] The present invention provides a tunnel leakage global positioning method based on a contour enhancement segmentation network of a panoramic-infrared dual camera. The specific process framework of the method of the present invention for panoramic detection and precise positioning of tunnel leakage areas is as follows: Figure 1 As shown. It is proposed to use dual fisheye cameras to capture tunnel panoramic images and perform tunnel wall segmentation, use the CEIL-Net model to accurately segment the infrared leakage area, and calculate the panoramic mapping of the leakage area based on the homography matrix of the camera posture to obtain the precise positioning of the leakage position. Specifically, it includes the following steps:

[0072] S1. Collect 360-degree panoramic images of the tunnel through dual fisheye cameras and collect thermal images of the leakage area through thermal infrared cameras;

[0073] S2, using the equidistant cylindrical mapping method to process the 360-degree panoramic image of the tunnel collected by the dual fisheye cameras, and converting the highly distorted spherical image into a planar panoramic image;

[0074] S3, separating the tunnel wall from the non-wall elements in the planar panoramic image, and expanding the tunnel wall into a rectangular image;

[0075] S4. Construct an infrared image leakage segmentation network based on CEIL-Net, take the thermal imaging of the leakage area as input, and the edge recognition result of the leakage area as output to train the infrared image leakage segmentation network and obtain the leakage area segmentation model;

[0076] S5. Input the thermal imaging of the leakage area into the leakage area segmentation model to obtain the edge recognition result of the leakage area; then use the panoramic mapping method of the leakage area based on the camera posture to map the edge recognition result of the leakage area to the plane image of the tunnel to achieve accurate positioning of the leakage position.

[0077] As a preferred embodiment, the specific principle steps of step S2 are as follows: the collected dual fisheye images contain all-round visual information of the tunnel, but due to its high distortion, it needs to be processed by a specific mapping method to convert it into a practical panoramic image. Equidistant cylindrical mapping is an effective image expansion method, which simulates the longitude and latitude expansion of the earth's surface and converts the spherical image into a plane image. Figure 2 As shown in Figure (a), a three-dimensional rectangular coordinate system is established with the optical center of the lens as the origin, where the XYZ axes satisfy the right-hand rule. Let the X axis coincide with the optical axis of the lens and point to the outside of the camera; the Z axis points to the top of the lens; and the Y axis points to the right side of the lens. Then the longitude of any point P in space projected along the straight line OP to the point P1 on the unit sphere can be expressed as the angle between the straight line OP projected onto the XOY plane and the X axis, and the commonly used symbol is Indicates that the left side of the X axis satisfy: On the right satisfy: The latitude of point P can be expressed as the angle between the straight line OP and the Z axis, usually represented by the symbol θ, where 0≤θ≤π / 2 is above the XOY plane and -π / 2≤θ≤0 is below it. Since the horizontal field of view (FOV) is 2π after the sphere is unfolded, and the vertical field of view is π, the aspect ratio of the panoramic image is usually 2:1. Under this mapping strategy, the ordinate of the image is converted into latitude, and the abscissa is converted into longitude. Specifically, if the width of the panoramic image is set to W1 and the height is set to H1, the longitude and latitude coordinates of any point (i, j) in the image, where i and j represent the pixel position of the point in the horizontal and vertical directions, respectively, can be obtained by converting the following mathematical formula:

[0078]

[0079] By mapping the dual fisheye images latitude and longitude onto a common plane, we are able to generate a seamless 360-degree panorama, such as Figure 2As shown in (b), the image retains the comprehensive visual information inside the tunnel while minimizing distortion.

[0080] As a preferred embodiment, refer to Figure 3 ,Step S3 is as follows: effectively separate the tunnel wall part from other non-wall elements in the tunnel panorama, and expand the tunnel wall into a rectangular image. The specific expansion principle is as follows Figure 3 As shown in (a). A three-dimensional Cartesian world coordinate system is established at the center of the tunnel, with the origin at the center of the tunnel bottom, the X-axis along the tunnel, the Y-axis perpendicular to the tunnel bottom, and the Z-axis vertically upward. Assuming that the radius of the tunnel is R, the length of the tunnel section to be unfolded is 2R, and the unfolded length of the tunnel wall is the circumference of the semi-cylinder πR. Therefore, the unfolded diagram of the tunnel wall forms a rectangle with a size of πR×2R. On this basis, a canvas with an aspect ratio of W2:H2=2×π is generated to simulate the image of the unfolded tunnel wall. In order to unfold the tunnel wall into a planar image, each pixel point on the canvas is first traversed. For each pixel point (μ, ν) on the canvas, the corresponding world coordinate P(x, y, z) on the tunnel wall is calculated according to its pixel coordinates. The conversion relationship is as follows:

[0081]

[0082] Next, a spherical coordinate system is established at the origin of the world coordinate system to simulate the shooting position of the dual fisheye panoramic camera in the center of the tunnel, and then the world coordinates are converted into spherical coordinates.

[0083]

[0084] Then, the equidistant cylindrical mapping principle is used to determine the corresponding position (i, j) of the pixel (μ, ν) on the canvas in the panoramic image. Finally, the color value of the corresponding pixel in the panoramic image is assigned to the pixel on the canvas to complete the image expansion of the tunnel wall. Through step S3, the tunnel wall part is effectively expanded into a rectangular image, such as Figure 3 As shown in (b).

[0085] As a preferred embodiment, the principle and steps of step S4 are as follows: the edge of the leakage area in the infrared image is too close to the surrounding environment, which makes it difficult for the traditional U-Net model to accurately segment the fine leakage area; secondly, in the deep network, due to continuous pooling and convolution operations, important edge feature information may be lost. In order to solve the above problems, the present invention improves the U-Net network and proposes a new network architecture CEIL-Net to improve the current situation of the unsatisfactory segmentation effect of U-Net. The structural framework of the model is as follows Figure 4As shown. Specifically, the contour filter kernel (OLK) is introduced in the cross-layer connection of the first two layers of CEIL-Net. By significantly enhancing the edge features in the image in the shallow convolution layer, it helps the model better capture the contours of the leakage area, thereby improving the problem of blurred edges. In order to further enhance the feature selection and recalibration capabilities of the network, a Squeeze-and-Excitation (SE) module is added after every two convolution operations of CEIL-Net. This module significantly improves the network's ability to enhance key features and suppress non-key features by learning the dependencies between feature channels. In order to comprehensively consider all aspects of segmentation performance, CEIL-Net adopts a combined loss function that combines cross entropy loss, Dice loss, and edge-aware loss. This combined loss function can not only deal with the problem of class imbalance, but also strengthen the accuracy requirements for segmentation edges.

[0086] Aiming at the edge blur problem in the infrared tunnel leakage segmentation task, a new contour convolution kernel is introduced. The contour convolution kernel is designed with positive weights for the central pixels and negative weights for the surrounding pixels. This design can not only effectively extract the contour information of the image, but also reduce the impact of edge blur on the segmentation accuracy.

[0087]

[0088] After every two convolution operations in the model, a (SE) module is added to further extract the channel information of the feature map. The structure of the SE module is as follows Figure 5 shown.

[0089] This model improves model performance by combining cross entropy loss, Dice loss and edge-aware loss. Cross entropy loss is suitable for pixel-level classification tasks and can guide the model to learn overall features; Dice loss handles unbalanced data distribution and emphasizes target segmentation accuracy; edge-aware loss is designed for edge blur to improve segmentation accuracy and robustness. Its specific definition is as follows:

[0090] L total is the total combined loss function, defined as the cross entropy loss L CE , Dice loss L Dice , and edge-aware loss L Edge The weighted sum of:

[0091] L total =α·L CE +β·L Dice +γ·L Edge

[0092] Among them, α, β, and γ are weight coefficients used to adjust the contribution of different loss functions to the total loss.

[0093]

[0094] Where N is the total number of pixels, y i is the true label of the i-th pixel, p i is the probability that the pixel is predicted to be positive, i.e., leakage.

[0095] As a preferred embodiment, step S5 is as follows: the first step is to project the spherical content of the dual fisheye camera onto a cube model, and then expand it on each surface. Since the tunnel wall is to be mapped, only images of the left, right and upper three projection surfaces are taken.

[0096] After the panoramic image is stereoscopically segmented, the projection images in the left, top, and right directions are obtained. The next step is to map the stereoscopic projection image to the infrared camera under the same viewing angle. Therefore, the two cameras need to be calibrated together. In order to accurately calibrate the dual-modal camera system, this study uses a special thermal infrared calibration plate. By heating the calibration plate to a certain temperature, capturing a certain number of images with two-modal cameras at the same time, and then calibrating it using the Zhang Zhengyou calibration method, the internal parameters of each camera, K, are obtained. vis For panoramic cameras, K ir For thermal infrared cameras. External parameters, R vis , T vis For panoramic cameras, and R ir , T ir For thermal infrared cameras. In order to project the leakage area of ​​the infrared image onto the tunnel panoramic image, the present invention needs to solve the homography matrix between the two cameras, such as Figure 6 As shown, let the world coordinate system W and the camera coordinate system x1y1z1 of the thermal infrared camera coincide with each other, and calculate the relative position of the panoramic camera relative to the thermal infrared camera based on the calibrated internal and external parameters of the two cameras as [R relative T relative ], and calculate the homography matrix H according to the above camera posture.

[0097]

[0098] Finally, according to Figure 3 The segmentation principle is used to project the leakage area in the panoramic segmentation image onto the tunnel wall expansion image. The conversion relationship between the point coordinates (x, y, z) of the leakage area in the panoramic segmentation image in the O-XYZ world coordinate system and the pixel coordinates (μ, ν) of the tunnel wall expansion image is:

[0099]

[0100] Where ψ is the angle between the projection line of the point on the segmented image and the line connecting point O in the O-YZ plane and the Z axis.

[0101]

[0102] in, and Represent the gradients of the true label and the predicted label at the i-th pixel respectively.

[0103] refer to Figure 1 , the present invention also provides a tunnel leakage global positioning system based on a contour enhancement segmentation network of a panoramic-infrared dual camera, including: a camera system, a camera fixing and adjustment mechanism, a mobile bracket system, a power management module, a data transmission module, and a data processing module; the camera system includes a dual fisheye camera and an infrared camera;

[0104] The dual fisheye camera uses two 180-degree fisheye lenses to shoot at the left and right sides of the tunnel respectively, so as to capture a 360-degree panoramic image; the focal length of each fisheye camera is 7.2 mm, and a 1 / 2-inch CMOS photosensitive element is used;

[0105] The infrared camera uses an uncooled focal plane thermal infrared detector with a focal length of 3.2 mm, a field of view of 55.6°×42°, and a resolution of 256×192, which is used to capture thermal images inside the tunnel;

[0106] The camera fixing and adjusting mechanism is used to install the dual fisheye camera and the infrared camera on the multifunctional mobile trolley. The camera is placed on the top layer, and a mobile power supply and a computer are configured on the bottom layer. The computer is used to control the camera's shooting screen in real time, save the collected picture data and process it;

[0107] The power management module is used to provide continuous power to the camera system and the computer;

[0108] The data transmission module transmits the data collected by the camera system to the computer;

[0109] The data processing module is built into the computer and includes a plane panorama conversion unit, a rectangular image generation unit, a leakage area segmentation model training unit, and a leakage position positioning unit;

[0110] The panoramic image conversion unit is used to process the 360-degree panoramic image of the tunnel collected by the dual fisheye cameras using an equidistant cylindrical mapping method, and convert the highly distorted spherical image into a flat panoramic image;

[0111] A rectangular image generating unit, used for separating the tunnel wall from non-wall elements in the planar panoramic image, and expanding the tunnel wall into a rectangular image;

[0112] The leakage area segmentation model training unit is used to build an infrared image leakage segmentation network based on CEIL-Net, and uses the leakage area thermal imaging as input and the edge recognition result of the leakage area as output to train the infrared image leakage segmentation network and obtain the leakage area segmentation model;

[0113] The leakage position positioning unit is used to input the thermal imaging of the leakage area into the leakage area segmentation model to obtain the edge recognition result of the leakage area; then, the edge recognition result of the leakage area is mapped onto the plane image of the tunnel using the panoramic mapping method of the leakage area based on the camera posture to achieve accurate positioning of the leakage position.

[0114] The system of the present invention has the following advantages: (1) In view of the problem that it is difficult to fully display the location of internal defects in tunnel disease detection, it is proposed to capture 360-degree panoramic images through a dual fisheye camera configuration, use an equidistant cylindrical mapping method to process the images, convert the fisheye images into practical panoramic images, and use a tunnel wall segmentation method from a cylindrical image to a flat image to achieve a planar display of the tunnel wall, thereby overcoming the limitations of the viewing angle of traditional cameras and the complexity of image data management;

[0115] (2) Aiming at the problem of inaccurate image segmentation caused by low resolution of infrared images and low edge clarity of leakage areas, a new network model CEIL-Net is developed by improving the U-Net network model. The contour filter kernel and SE attention mechanism, as well as the combined loss function, are introduced to significantly enhance the edge features in the image and improve the ability to recognize the edge of the leakage area, thereby improving the limitations of the traditional model in the infrared image leakage segmentation task;

[0116] (3) In view of the fact that existing projection methods are mainly used for images with rich texture features, while infrared image projection faces challenges, a dual-mode camera system with integrated panoramic and thermal infrared functions was designed and built, and accurately calibrated using a special thermal infrared calibration plate. The homography matrix was directly solved using the known camera posture, and the leakage area segmented in the infrared image was accurately mapped to the tunnel wall unfolded image. The leakage area was marked with a highlight color, achieving comprehensive visualization of the leakage location.

[0117] The experimental supporting project of the present invention is a long-term safety entity test field for tunnel engineering located in Qingyuan City, Guangdong Province. It is part of the Huiqing section of the Shantou-Zhanjiang Expressway, with a total length of 579 meters. The cross-section of the tunnel is a circle with a diameter of 9.5 meters. A small part of the circle is underground, and the center of the circle is located 1.7 meters above the ground. The tunnel is located within the Taihe Gudong landslide body, with poor surrounding rock stability and abundant groundwater. Therefore, there is a large amount of water leakage inside the tunnel, which is suitable for the application of the method in this article.

[0118] In order to realize the projection mapping from infrared image to panoramic segmentation image, the dual-camera system should be jointly calibrated at five directions, namely 0°, 45°, 90°, 135°, and 180°, using a special thermal infrared calibration plate. Figure 7 As shown. The height of the tripod of the dual fisheye camera is set to 1.7 meters, so that the panoramic camera on the tripod is located at the center of the tunnel section. The pan / tilt is adjusted so that the infrared camera's line of sight shoots the right wall of the tunnel horizontally, and this direction is regarded as the 0° direction for calibration. First, the calibration plate is fixed on the right wall of the tunnel, and the panoramic camera is used to shoot the panoramic image, and then the infrared camera is used to shoot the infrared image. The calibration plate is removed, and the calibration plate in different postures is shot with two cameras. A total of 24 sets of infrared images and panoramic images are obtained, and the panoramic images are projected into cubes. The technical parameters of the projection image are: resolution 1000×1000, and the field of view angle of each face reaches 90×90 degrees. 24 cube projection images containing the calibration plate are obtained. According to these images, the Zhang Zhengyou calibration method is used to obtain the internal and external parameters of the dual fisheye camera and infrared camera in the dual camera system in the 0° direction. Keep the dual fisheye cameras fixed, use the gimbal to control the infrared camera to rotate 45° counterclockwise vertically, and obtain the camera parameters in the 45° direction in the same way as before; similarly, obtain the camera parameters of the dual camera system at 90°, 135°, and 180°, and calculate the single-image matrix in five directions according to the calculation method described above. The specific values ​​are shown in Table 1.

[0119] Table 1 Single-image matrix values

[0120]

[0121] After the calibration is completed, it is necessary to collect panoramic images and infrared images of the tunnel. The best environmental condition for infrared thermal imaging to collect tunnel leakage is after rain, so there will be more water leakage inside the tunnel. The tunnel detection section selected in this paper is 48 meters long, with 12 detection points arranged, each detection point is 4 meters apart, and the specific arrangement is as follows Figure 8 As shown in the figure, the dual-mode camera system is fixed in the middle of the tunnel, the tripod height is adjusted to 1.7 meters, and a panoramic image is taken with a panoramic camera. The infrared dual-light camera is rotated vertically by 0°, 45°, 90°, 135°, and 180° by rotating the camera gimbal to take infrared images and visible light images at five different angles. Then, after the dual-camera system moves 4 meters along the tunnel direction, the dual-camera system is fixed and panoramic images and infrared images are taken in the same steps. According to statistics, a total of 12 panoramic images and 60 infrared images are taken. These images are used as data set 1, which is mainly used for mapping the leakage area of ​​the tunnel. In order to train the proposed CEIL-Net model, infrared images are used to randomly shoot leakage areas at different angles and distances, and 2317 infrared images are obtained as data set 2, which are mainly used for training the proposed model.

[0122] The first step of data processing is to expand the two fisheye images collected by the dual fisheye cameras in data set 1. The expanded image size is 6528*3264. The tunnel wall is segmented using the segmentation method mentioned above. The segmented tunnel wall image size is 7690*3264, which is a schematic diagram of the expansion of the three shooting points.

[0123] The CEIL-Net model for segmenting infrared leakage areas is trained so that the model can automatically segment leakage contours from infrared images. Establishing a dataset is the basis for applying deep learning methods. Before testing, a large number of leakage images of various parts of the tunnel at different times and under different lighting conditions were collected using infrared cameras, and a dataset containing 2317 images was obtained2. The training set was labeled manually, and the labelme tool was used to draw the outline of the leakage area from the picture. The annotation information will be saved in the specified folder in the form of a JSON file, including the labeled category, attributes, and contour information. After manual labeling, the image is enhanced by adding rotation, random noise, random illumination, and random shadows, and finally 5799 training images are obtained. By counting the size of the marker box in the image, it is found that the dataset includes a total of 6138 marker boxes. Before using the dataset to train the model, the dataset needs to be randomly divided into two groups, 80% as the training set and 20% as the test set.

[0124] In order to verify the effectiveness of the contour enhancement model proposed in this paper, the aforementioned CEIL-Net model is trained using the established data set. To evaluate the effectiveness of the proposed model, this paper uses three state-of-the-art models, FCN, U-Net, and DeeplabV3+, to conduct comparative experiments with the proposed model. In order to carry out the test, a PC with Windows 10 operating system, a processor (CPU) model of Intel i7 7 700, 8 cores, a memory (RAM) of 8GB, and a graphics card (GPU) of NVIDIA GeForce GTX1050 is used as the operating environment. It is built based on Python language and pytorch framework, Pycharm is selected as IDE (integrated development environment), Anaconda is used as the interpreter, and the dependency packages required for the project are installed to complete the training, verification and testing of the model. SGD is used as the optimizer, the maximum learning rate Init_lr is set to 7e-3, the minimum learning rate defaults to 0.01 of the maximum learning rate, the number of training iterations is set to 200 times, and the number of batches is set to 4. Fig. 9The loss curve and miou curve of the two models during training. The training results show that the Miou of CEIL-Net is 88.02%, and the Miou of the previous segmentation model DeeplabV3+ is 85.034, proving that after adding contour enhancement and Squeeze-and-Excitation attention mechanism and using the combined loss function, the segmentation effect of the model is improved by 2.989%. Four infrared images containing leakage are randomly selected for testing. The images are input into the trained segmentation model for tunnel leakage segmentation.

[0125] right Figure 8 The three measuring point data shown in the figure are processed, and the panoramic image is mapped to obtain the unfolded image of the tunnel wall. After the infrared image is processed by the CEIL-Net model, the outline of the leakage area is accurately marked. We use the homography matrix calculated by the camera calibration parameters. The homography matrices in five directions are shown in Table 1. The leakage areas in these infrared images are mapped to the unfolded tunnel wall image. The highlight display technology is used in the mapping process to clearly mark each leakage point in the panoramic image of the tunnel wall. The mapping results are shown in Fig.10 The experimental results show that the mapping accuracy of the leakage area has reached more than 80%, and there is almost no missed detection or false detection during the mapping process. In addition, the mapping positioning accuracy between the panoramic image and the infrared image is within 10cm, which provides highly reliable reference data for future tunnel maintenance work.

[0126] Although the present invention has been described above with preferred embodiments, it is not intended to limit the present invention. A person skilled in the art of the present invention may make various modifications and improvements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the definition of the claims.

Claims

1. A tunnel leakage global positioning method based on a contour enhancement segmentation network of panoramic-infrared dual cameras, characterized in that: The following steps are involved: S1. Collect 360-degree panoramic images of the tunnel through dual fisheye cameras and collect thermal images of the leakage area through thermal infrared cameras; S2, using the equidistant cylindrical mapping method to process the 360-degree panoramic image of the tunnel collected by the dual fisheye cameras, and converting the highly distorted spherical image into a planar panoramic image; S3, separating the tunnel wall from the non-wall elements in the planar panoramic image, and expanding the tunnel wall into a rectangular image; S4. Construct an infrared image leakage segmentation network based on CEIL-Net, take the thermal imaging of the leakage area as input, and the edge recognition result of the leakage area as output to train the infrared image leakage segmentation network and obtain the leakage area segmentation model; S5. Input the thermal imaging of the leakage area into the leakage area segmentation model to obtain the edge recognition result of the leakage area; then use the panoramic mapping method of the leakage area based on the camera posture to map the edge recognition result of the leakage area to the plane image of the tunnel to achieve accurate positioning of the leakage position.

2. According to claim 1, a tunnel leakage global positioning method based on a contour enhancement segmentation network of a panoramic-infrared dual camera is characterized in that: In step S2, the equidistant cylindrical mapping method includes the following sub-steps: S2.

1. Establish a three-dimensional rectangular coordinate system with the optical center of the lens as the origin, where the XYZ axes satisfy the right-hand rule, and the X axis coincides with the optical axis of the lens and points to the outside of the camera; The Z axis points to the top of the lens; the Y axis points to the right side of the lens; S2.

2. For any point P in space, the longitude of point P1 projected onto the unit sphere along the straight line OP is expressed as: The angle between the straight line projected onto the XOY plane and the X-axis is: The left side of the X axis satisfy: On the right satisfy: The latitude of point P is expressed as the angle θ between the straight line OP and the Z axis, where θ above the XOY plane satisfies: 0≤θ≤π / 2, and θ below the XOY plane satisfies: -π / 2≤θ≤0; S2.

3. After the spherical image is expanded, the horizontal field of view is 2π, and the vertical field of view is π, that is, the aspect ratio of the panoramic image is 2:

1. The ordinate of the image is converted into latitude, and the abscissa is converted into longitude. The longitude and latitude of the dual fisheye images are mapped to a common plane. Let the width of the panoramic image be W1 and the height be H1. The longitude and latitude coordinates of any point (i, j) in the image are obtained by the following conversion formula: Where i and j represent the pixel position of the point in the horizontal and vertical directions respectively.

3. According to claim 2, a tunnel leakage global positioning method based on a contour enhancement segmentation network of a panoramic-infrared dual camera is characterized in that: Step S3 includes the following sub-steps: S3.

1. Establish a three-dimensional Cartesian world coordinate system at the center of the tunnel, with the origin at the center of the tunnel bottom, the X-axis along the tunnel, the Y-axis perpendicular to the tunnel bottom, and the Z-axis vertically upward; the radius is R, the length of the tunnel section to be unfolded is 2R, and the unfolded length of the tunnel wall is the circumference of the semi-cylinder πR, so the unfolded view of the tunnel wall forms a rectangle with a size of πR×2R, and then generate a canvas with an aspect ratio of W2:H2=2×π to simulate the image of the tunnel wall after unfolding; S3.

2. Traverse each pixel point on the canvas. For each pixel point (μ, ν) on the canvas, calculate the corresponding world coordinate P (x, y, z) on the tunnel wall according to its pixel coordinates. The conversion relationship is as follows: S3.

3. Establish a spherical coordinate system at the origin of the world coordinate system to simulate the shooting position of the dual fisheye panoramic camera in the center of the tunnel, and then convert the world coordinates into spherical coordinates S3.

4. Use the equidistant cylindrical mapping principle to determine the corresponding position (i, j) of the pixel point (μ, ν) on the canvas in the panoramic image, assign the color value of the corresponding pixel in the panoramic image to the pixel on the canvas, and realize the expansion of the image of the tunnel wall into a plane image.

4. The tunnel leakage global positioning method based on the contour enhancement segmentation network of panoramic-infrared dual cameras according to claim 1 is characterized in that: Step S4 performs image leakage segmentation based on the CEIL-Net network, including the following sub-steps: S4.

1. The contour filter kernel OLK is introduced in the cross-layer connection of the first two layers of the CEIL-Net network. By significantly enhancing the edge features in the image in the shallow convolution layer, the SE module is added after every two convolution operations of CEIL-Net; and the cross entropy loss, Dice loss and edge-aware loss are combined into a total combined loss function to improve the model performance; S4.2, the contour filter kernel OLK takes the weight of the central pixel as positive and the weight of the surrounding pixels as negative, as shown in the following formula: S4.

3. After every two convolution operations in the model, a (SE) module is added to further extract the channel information of the feature map. S4.

4. The total combined loss function is as follows: L total =α·L CE +β·L Dice +γ·L Edge , Among them, L CE is the cross entropy loss, L Dice is the Dice loss, L Edge is the edge-aware loss, α, β, γ are weight coefficients used to adjust the contribution of different loss functions to the total loss; In the formula, Where N is the total number of pixels, y i is the true label of the i-th pixel, p i is the probability that the pixel is predicted as positive, i.e. leakage, ▽y i and ▽p i Represent the gradients of the true label and the predicted label at the i-th pixel respectively.

5. The tunnel leakage global positioning method based on the contour enhancement segmentation network of panoramic-infrared dual cameras according to claim 2 is characterized in that: In step S5, the panoramic mapping method of the leakage area based on the camera posture includes the following sub-steps: S5.1, stereoscopically segmenting the panoramic unfolded image to obtain projection images in three directions of left, top and right; S5.2, a thermal infrared calibration plate was used, the calibration plate was heated to a preset temperature, and a preset number of images were captured using two-mode cameras at the same time, and then calibrated using the Zhang Zhengyou calibration method to obtain the internal parameters of each camera, including: K vis For panoramic cameras, K ir For thermal infrared cameras; external parameters include: R vis , T vis For panoramic cameras, and R ir , T ir For thermal infrared cameras; S5.

3. Solve the homography matrix between the two cameras, let the world coordinate system W and the camera coordinate system x1y1z1 of the thermal infrared camera coincide, and calculate the relative position of the panoramic camera relative to the thermal infrared camera based on the calibrated internal and external parameters of the two cameras as [R relative T relative ], and calculate the homography matrix H according to the above camera posture. S5.

4. The leakage area in the panoramic segmentation image is projected onto the tunnel wall expansion image. The conversion relationship between the point coordinates (x, y, z) of the leakage area in the panoramic segmentation image in the O-XYZ world coordinate system and the pixel coordinates (μ, ν) of the tunnel wall expansion image is: Where ψ is the angle between the projection line of the point on the segmented image and the line connecting point O in the O-YZ plane and the Z axis.

6. A tunnel leakage global positioning system based on a contour enhancement segmentation network of panoramic-infrared dual cameras, characterized in that: include: Camera system, camera fixing and adjustment mechanism, mobile bracket system, power management module, data transmission module, data processing module; The camera system includes dual fisheye cameras and infrared cameras; The dual fisheye camera uses two 180-degree fisheye lenses to shoot at the left and right sides of the tunnel respectively, so as to capture a 360-degree panoramic image; the focal length of each fisheye camera is 7.2 mm, and a 1 / 2-inch CMOS photosensitive element is used; The infrared camera uses an uncooled focal plane thermal infrared detector with a focal length of 3.2 mm, a field of view of 55.6°×42°, and a resolution of 256×192, which is used to capture thermal images inside the tunnel; The camera fixing and adjusting mechanism is used to install the dual fisheye camera and the infrared camera on the multifunctional mobile trolley. The camera is placed on the top layer, and a mobile power supply and a computer are configured on the bottom layer. The computer is used to control the camera's shooting screen in real time, save the collected picture data and process it; The power management module is used to provide continuous power to the camera system and the computer; The data transmission module transmits the data collected by the camera system to the computer; The data processing module is built into the computer and includes a plane panorama conversion unit, a rectangular image generation unit, a leakage area segmentation model training unit, and a leakage position positioning unit; The panoramic image conversion unit is used to process the 360-degree panoramic image of the tunnel collected by the dual fisheye cameras using an equidistant cylindrical mapping method, and convert the highly distorted spherical image into a flat panoramic image; A rectangular image generating unit, used for separating the tunnel wall from non-wall elements in the planar panoramic image, and expanding the tunnel wall into a rectangular image; The leakage area segmentation model training unit is used to build an infrared image leakage segmentation network based on CEIL-Net, and uses the leakage area thermal imaging as input and the edge recognition result of the leakage area as output to train the infrared image leakage segmentation network and obtain the leakage area segmentation model; The leakage position positioning unit is used to input the thermal imaging of the leakage area into the leakage area segmentation model to obtain the edge recognition result of the leakage area; then, the edge recognition result of the leakage area is mapped onto the plane image of the tunnel using the panoramic mapping method of the leakage area based on the camera posture to achieve accurate positioning of the leakage position.

Images