Method for 3d presentation of pipeline inspection videos

Abstract

This application relates to the field of geographic information technology and discloses a method for 3D display of pipeline inspection videos, comprising: acquiring a 3D model of a target pipeline and an inspection video of the interior of the target pipeline; correcting the pipeline center point and pipeline cross-section for each frame of the inspection video to obtain corrected video frame images; acquiring a panoramic image of the interior of the target pipeline based on each corrected video frame image; mapping the pixels in the panoramic image to the 3D model of the pipeline to generate a textured 3D model of the pipeline; and displaying the textured 3D model of the pipeline in a preset 3D scene. This method improves the accuracy of displaying the internal condition of the pipeline in a 3D scene. This application also discloses an apparatus, electronic device, and storage medium for 3D display of pipeline inspection videos.

Description

Technical Field

[0001] This application relates to the field of geographic information technology, for example to a method for 3D video display of pipeline inspection. Background Technology

[0002] Pipelines, as crucial infrastructure, are widely used in water supply and drainage, oil and gas transportation, and industrial fluid transport. With increasing service life, pipelines are prone to defects such as cracks, corrosion, deformation, and blockages. If not inspected and repaired in a timely manner, these defects can lead to safety accidents such as leaks and collapses. Therefore, pipeline inspection is a vital aspect of urban pipeline network operation and maintenance management. With the increasing demands for digital management of underground pipelines, how to intuitively and accurately display the internal condition of pipelines has become a key focus in this field.

[0003] Currently, CCTV (Closed Circuit Television) inspection technology is the mainstream method for inspecting the interior of pipelines. Inspection robots equipped with cameras enter the pipeline, capture images of its interior, and transmit them to a ground terminal. Technicians then view the video to identify defects (such as cracks, deformations, and siltation). However, traditional inspection results are typically presented as two-dimensional video images, lacking spatial correlation and offering a less intuitive presentation. While methods exist to combine 3D pipeline models with inspection videos, perspective distortion occurs when CCTV cameras capture images inside the pipeline, and the camera's movement cannot always be kept centered. This results in poor alignment between the video image and the 3D pipeline model, leading to texture distortion and misalignment in the generated 3D pipeline model, resulting in unsatisfactory display effects.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0006] This disclosure provides a method, apparatus, electronic device, and storage medium for 3D video display of pipeline inspection, which can improve the display accuracy of the internal condition of the pipeline in a 3D scene.

[0007] In some embodiments, the method includes: acquiring a 3D model of the target pipe and a detection video of the inside of the target pipe; performing pipe center point correction and pipe cross-section correction on each frame of the detection video to obtain corrected video frame images; acquiring a panoramic image of the inside of the target pipe based on each corrected video frame image; mapping the pixels in the panoramic image to the 3D model of the pipe to generate a textured 3D model of the pipe; and displaying the textured 3D model of the pipe in a preset 3D scene.

[0008] In some embodiments, the apparatus includes: a modeling module configured to acquire a 3D model of a target pipe; a detection video acquisition module configured to acquire a detection video of the interior of the target pipe; an image correction module configured to perform pipe center point correction and pipe cross-section correction on each frame of the detection video to obtain a corrected video frame image; a panoramic image acquisition module configured to acquire a panoramic image of the interior of the target pipe based on each corrected video frame image; a model mapping module configured to map pixels in the panoramic image to the 3D model of the pipe to generate a textured 3D model of the pipe; and a display module configured to display the textured 3D model of the pipe in a preset 3D scene.

[0009] In some embodiments, the apparatus includes a processor and a memory storing program instructions, the processor being configured to execute the method described above for 3D video display of pipeline inspection when the program instructions are executed.

[0010] In some embodiments, the electronic device includes: an electronic device body; and the aforementioned device for 3D video display of pipeline inspection is mounted on the electronic device body.

[0011] In some embodiments, the storage medium stores program instructions that are executed by a processor to implement the above-described method for 3D video display of pipeline inspection.

[0012] The method, apparatus, electronic device, and storage medium for three-dimensional video display of pipeline inspection provided in this disclosure can achieve the following technical effects: By correcting the pipe center point and pipe cross-section of each frame of the video footage from inside the pipe, image distortion caused by positional offset and perspective distortion when the camera captures images inside the pipe can be effectively eliminated. This results in corrected video frame images with a unified viewpoint and corrected geometric deformation. Generating panoramic images based on these corrected video frame images significantly improves the generation accuracy and geometric consistency of the panoramic images. Mapping this panoramic image to a 3D pipe model improves the accuracy of texture mapping, ensuring that the generated textured 3D pipe model is precisely aligned with the actual pipe structure in space. This enhances the accuracy of displaying the pipe's internal conditions in a 3D scene.

[0013] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0014] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein: Figure 1 This is a schematic diagram of a method for three-dimensional video display of pipeline inspection provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram of a frame of CCTV detection video inside the target pipe provided in an embodiment of this disclosure; Figure 3 This is a schematic diagram of another frame of CCTV detection video inside the target pipe provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of a corrected video frame image provided in an embodiment of this disclosure; Figure 5 This is a schematic diagram of an effective annular image provided in an embodiment of this disclosure; Figure 6 This is a schematic diagram of a rectangular image provided in an embodiment of this disclosure; Figure 7 This is a schematic diagram of a panoramic image provided in an embodiment of this disclosure; Figure 8 This is a schematic diagram of the structure of a device for three-dimensional video display of pipeline inspection provided in an embodiment of this disclosure; Figure 9 This is a schematic diagram of another device for three-dimensional video display of pipeline inspection provided in an embodiment of this disclosure. Detailed Implementation

[0015] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0016] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0017] Unless otherwise stated, the term "multiple" means two or more.

[0018] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0019] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0020] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0021] The method for 3D display of pipeline inspection videos provided in this disclosure uses an electronic device as the execution subject. The electronic device includes a computer or server. The electronic device utilizes BIM (Building Information Modeling) technology to construct a 3D model of the target pipeline. The electronic device corrects the pipeline center point and pipeline cross-section in the CCTV inspection video of the target pipeline. The electronic device extracts valid annular images from each corrected video frame; uses an inverse polar coordinate algorithm to unfold each valid annular image into a rectangular image; and stitches the rectangular images together to obtain a panoramic image of the target pipeline's interior; finally, it applies texture mapping to the 3D pipeline model. The method for 3D display of pipeline inspection videos provided in this disclosure achieves accurate fusion of CCTV images and the 3D pipeline model, improving the 3D display effect of the pipeline inspection video. It is applicable to pipeline inspection in municipal, oil and gas, and other fields.

[0022] Combination Figure 1As shown in the embodiments of this disclosure, a method for three-dimensional display of pipeline inspection videos is provided, including: Step S101: The electronic device acquires the 3D model of the target pipe and the detection video of the inside of the target pipe.

[0023] In some embodiments, the target pipe is the pipe to be displayed. The 3D model of the pipe is a cylindrical mesh model, consisting of several vertices and triangular faces connecting the vertices.

[0024] Optionally, the electronic device acquires a 3D model of the target pipe, including: First, collect pipeline mapping data for the target pipeline. This data includes at least pipeline direction data, endpoint coordinate data, and elevation data to define the pipeline's spatial trajectory, as well as pipe diameter and material data to define the pipeline's physical properties. Pipeline mapping data can be obtained through on-site measurements, extraction from a geographic information system (GIS), or archiving of existing pipeline data.

[0025] Secondly, the collected pipeline mapping data is imported into BIM software, and a 3D solid model of the target pipeline is built based on this data. Specifically, based on the endpoint coordinate data and elevation data, the spatial coordinates of the starting and ending points of the target pipeline are determined in a 3D spatial coordinate system. Then, based on the pipeline route data, a pipeline centerline trajectory connecting the starting and ending points is fitted and generated. Based on the pipe diameter and material data, corresponding pipeline component templates are selected from the parametric component library, and a 3D solid model with corresponding geometric dimensions and physical properties is generated using a sweep forming algorithm along the pipeline centerline trajectory. This 3D solid model accurately reflects the spatial location, geometric shape, and material characteristics of the target pipeline. Furthermore, utilizing the built-in measurement function of the modeling software, the length L and diameter D of the target pipeline are automatically extracted from the attributes of the 3D solid model.

[0026] Finally, the 3D solid model is discretized into a mesh to obtain the 3D model of the target pipeline. Specifically, the geometric surface of the 3D solid model is discretized into several vertices and triangular facets formed by connecting these vertices using a surface triangulation algorithm, thereby generating a cylindrical mesh model corresponding to the target pipeline, i.e., the 3D model of the pipeline. The size of the triangular facets can be adaptively adjusted according to accuracy requirements.

[0027] In some embodiments, the inspection video inside the target pipe is a CCTV inspection video of the target pipe's interior. Video of the target pipe's interior is acquired using a pipe closed-circuit television (CCTV) inspection system. The system determines whether the acquired video contains images of the pipe's interior and / or exterior. If so, a preset video processing tool is used to delete these images, resulting in the CCTV inspection video of the target pipe's interior. If not, the acquired video is identified as the CCTV inspection video of the target pipe's interior. This CCTV inspection video records image information of the target pipe's inner wall. The preset video processing tool may be LosslessCut, CapCut, or Adobe Premiere Pro, etc.

[0028] "Not included in the pipeline view" refers to a video frame image in which the imaging width of the pipeline's near-end cross-section is less than 90% of the width of the video frame, or the imaging height of the pipeline's near-end cross-section is less than 90% of the height of the video frame. The pipeline's near-end cross-section is the cross-section of the pipeline closest to the camera along its axial direction.

[0029] "Out-of-view pipeline footage" refers to a video frame image in which the imaging width of the pipeline's distal cross-section is greater than 50% of the width of the video frame, or the imaging height of the pipeline's distal cross-section is greater than 50% of the height of the video frame. The distal cross-section is the cross-section of the pipeline furthest from the camera along its axial direction.

[0030] Optionally, after acquiring the CCTV inspection video inside the target pipe, the process further includes: uniformizing the playback speed of the CCTV inspection video to ensure a consistent playback speed.

[0031] In some embodiments, a preset video processing tool is used to uniformize the speed of CCTV inspection videos.

[0032] Optionally, the CCTV inspection video is subjected to speed-regulating processing, including: using the camera movement speed detection function built into the video processing tool, or a camera movement speed detection algorithm based on feature point detection, detecting the camera movement speed in the CCTV inspection video at preset time intervals, marking segments with excessively slow and / or excessively fast speeds, and recording the speed multiple of each segment relative to the average speed. The average speed is obtained by dividing the length (in meters) of the target pipe corresponding to the acquired CCTV inspection video by the total duration (in seconds) of the CCTV inspection video, with units of meters per second. This average speed reflects the theoretical uniform travel speed of the camera throughout the entire inspection process. A preset duration of 1 second improves computational efficiency and ensures detection accuracy.

[0033] Obtain the target duration for each interval segment, and use the built-in speed adjustment function of the video processing tool to adjust the duration of each interval segment to the corresponding target duration. This involves calculating T... i，target = T i，original / k i Obtain the target duration T for each interval segment. i，target Let T be the target duration of the i-th interval. i，original Let k be the original duration of the i-th interval. i Let be the speed multiplier for the i-th interval. For example, if the speed multiplier for a certain interval is 1.2 (meaning the actual speed of that interval is 20% faster than the average speed), then its target duration should be 0.83 times the original duration, which means the video of that interval needs to be compressed to 83% of its original length; if the speed multiplier is 0.8, then the target duration should be 1 / 0.8 = 1.25 times the original duration, which means the video of that interval needs to be stretched to 125% of its original length.

[0034] The video processing tool uses a frame smoothing algorithm to smooth the video speed according to the target duration of each segment, resulting in a CCTV inspection video with uniform speed. The frame smoothing algorithm includes, but is not limited to, frame repetition, frame dropping, frame interpolation, or optical flow frame interpolation to ensure the continuity and smoothness of the video after speed changes, avoiding visual jumps or stuttering caused by sudden speed changes.

[0035] After the above-mentioned speed uniformization process, the camera movement speed in the output CCTV inspection video remains uniform, eliminating the speed variations caused by manual operation or equipment resistance, which facilitates subsequent applications such as pipeline defect identification, texture mapping, or 3D model mapping.

[0036] In some embodiments, the camera movement speed detection algorithm based on feature point detection specifically involves: extracting significant feature points in CCTV detection video frame images using a feature point detection algorithm (such as SIFT, ORB, or FAST); finding the corresponding position of the same feature point in two consecutive frames using optical flow or a feature point matching algorithm, and calculating the movement distance Δd (in pixels) of the feature point in the image coordinate system; and calculating the camera movement speed corresponding to the feature point based on the time interval Δt (in seconds) between adjacent frames. v =Δd / Δt.

[0037] By calculating: k i = v i / v0 obtains the speed multiplier, where v iLet vi be the camera's moving speed in the i-th interval, and v0 be the average speed. If the calculated camera moving speed is greater than the average speed, it is marked as too fast; if the calculated camera moving speed is less than the average speed, it is marked as too slow. For example, a speed multiplier of 1.2 indicates that the speed in this interval is 20% faster than the average speed, and a speed multiplier of 0.8 indicates that the speed is 20% slower than the average speed.

[0038] In step S102, the electronic device performs pipe center point correction and pipe cross-section correction on each frame of the inspection video inside the target pipe to obtain corrected video frame images. In one embodiment, the inspection video inside the target pipe is a CCTV inspection video that has been processed to achieve uniform speed.

[0039] In step S103, the electronic device acquires a panoramic image of the inside of the target pipe based on each corrected video frame image.

[0040] In step S104, the electronic device maps the pixels in the panoramic image to the 3D model of the pipe, generating a textured 3D model of the pipe. This textured 3D model of the pipe is obtained by attaching the panoramic image of the inside of the target pipe to the surface of the 3D model of the pipe through texture mapping. Therefore, this textured 3D model of the pipe not only presents the geometry of the target pipe in 3D space, but also presents the true color and surface details of the inner wall of the target pipe.

[0041] In step S105, the electronic device displays the textured 3D model of the pipe in a preset 3D scene. The preset 3D scene is a 3D virtual scene constructed on a preset display device.

[0042] The method for 3D display of pipeline inspection videos provided in this disclosure effectively eliminates image distortion caused by positional offset and perspective distortion when the camera captures images inside the pipeline by correcting the pipeline center point and pipeline cross-section of each frame of the pipeline inspection video. This results in corrected video frame images with a unified viewpoint and corrected geometric deformation. Generating panoramic images based on these corrected video frame images significantly improves the generation accuracy and geometric consistency of the panoramic images. Mapping this panoramic image to a 3D pipeline model improves the accuracy of texture mapping, ensuring that the generated textured 3D pipeline model is precisely aligned with the actual pipeline structure in space. This enhances the display accuracy of the pipeline's internal conditions in a 3D scene.

[0043] Optionally, pipe center point correction is performed on each frame of the inspection video inside the target pipe, including: performing the following processing on each frame of the inspection video inside the target pipe: Edge detection is performed on the frame image using the Canny edge detection operator to obtain several edge lines. The extension direction of the target pipe in the frame image is determined. Edge lines whose angle difference between their extension direction and the extension direction of the target pipe is less than a set threshold are identified as target edge lines. The coordinates of the intersection point of the extensions of any two target edge lines are calculated. The arithmetic mean of the x-coordinates and y-coordinates of all intersection points are calculated to obtain the average coordinates; these average coordinates are then used as the estimated coordinates of the pipe's center point. The coordinates of the center point of the frame image are obtained. Based on the center point coordinates of the frame image and the estimated coordinates of the pipe's center point, the pipe's center point is corrected to the center point of the frame image. The center point of the frame image is the geometric center in the frame image's pixel coordinate system.

[0044] In one embodiment, such as Figure 2 As shown, Figure 2 This is a schematic diagram of a single frame from a CCTV inspection video of the target pipe's interior. Because the CCTV camera is positioned low and the crawling robot shifts during movement, the center point of the pipe in the captured video is generally not centered in the image. By first correcting the pipe's center point to the image center, the accuracy and geometric consistency of the generated panoramic image can be significantly improved. Furthermore, it ensures that the generated textured 3D pipe model is precisely aligned spatially with the actual pipe structure.

[0045] In one embodiment, edge detection is performed on a frame image using the Canny edge detection operator to obtain several edge lines, including: converting the frame image into a grayscale image. The Canny edge detection operator is then used to perform edge detection on the grayscale image to obtain a binarized edge image. In this binarized edge image, pixels with a value of 1 constitute edge lines, used to characterize the structural features of the pipe's inner wall, including pipe contours, circumferential seams, longitudinal seams, cracks, or deformation boundaries.

[0046] In one embodiment, the extension direction of the target pipe in the frame image is detected by Hough transform, or determined based on the symmetry of the two edges of the target pipe. The target edge lines are those whose angle difference between their extension direction and the extension direction of the target pipe is less than a set threshold. For example, Figure 3 As shown, Figure 3 This is a schematic diagram of a single frame of CCTV video footage from inside the target pipe. Figure 3 The red edge line in the image is the target edge line.

[0047] Optionally, the pipe center point is corrected to the center point of the frame image based on the center point coordinates of the frame image and the estimated coordinates of the pipe center point, including: calculating Δ = (Δx, Δy) = (x... b -x f y b –yf Obtain the offset vector between the center point coordinates of the frame image and the estimated coordinates of the pipe center point, where Δ is the offset vector, (x... b , y b (x) represents the estimated coordinates of the pipe's center point. f , y f (x) represents the center point coordinates of the frame image. f = W / 2, y f = H / 2, where W is the width of the frame image and H is the height of the frame image. Using the four corner points of the frame image as source control points and the positions of each corner point after translation by the offset vector Δ as target control points, calculate the first perspective transformation matrix. Use this first perspective transformation matrix to transform the frame image, correcting the pipe center point to the center point of the frame image, thus completing the correction of the pipe center point.

[0048] Optionally, the pipe center point is corrected to the center point of the frame image based on the center point coordinates of the frame image and the estimated coordinates of the pipe center point, including: calculating Δ = (Δx, Δy) = (x... b -x f y b –y f Obtain the offset vector between the center point coordinates of the frame image and the estimated coordinates of the pipe center point, where (Δx, Δy) is the offset vector, (x... b , y b (x) represents the estimated coordinates of the pipe's center point. f , y f (x) represents the center point coordinates of the frame image. f = W / 2, y f = H / 2, where W is the width of the frame image and H is the height of the frame image. The frame image is then translated using this offset vector, i.e., each pixel (x, y) in the frame image is translated. o ,y o Transform to (x) o -Δx, y o -Δy) is used to align the center point of the pipe with the center point of the frame image, thus completing the correction of the pipe's center point.

[0049] Optionally, pipe cross-section correction is performed on each frame of the inspection video inside the target pipe, including: performing the following processing on each frame of the inspection video inside the target pipe: Edge detection is performed on the frame image using the Canny edge detection operator to obtain several edge lines. Edge lines forming closed contours are identified from these edge lines, and ellipse fitting is performed on each closed contour to obtain the corresponding ellipse parameters. The closed contour edge lines corresponding to the ellipse parameters that meet preset screening conditions are determined as elliptical edge contours representing the pipe cross-section. Based on the ellipse parameters of this elliptical edge contour, a perspective transformation algorithm is used to map the elliptical edge contour in the frame image to a circular edge contour. The ellipse parameters include: the coordinates of the ellipse center, the major semi-axis, the minor semi-axis, and the rotation angle.

[0050] The preset filtering conditions include one or more of the following: the ratio of the major and minor axes of the ellipse is within a preset range, for example, between 1.0 and 2.0; the area of ​​the ellipse is greater than a first preset threshold; and the distance between the center of the ellipse and the center of the frame image is less than a second preset threshold.

[0051] Because the CCTV inspection system's camera is positioned low, and the crawling robot may shift during movement, the pipe cross-section in the captured video frames is generally elliptical. By correcting the elliptical edge contour of the pipe cross-section to a circle, a foundation is laid for obtaining a relatively orthogonal graphic after subsequent inverse polar coordinate unfolding. This significantly improves the generation accuracy and geometric consistency of the panoramic image. Furthermore, it ensures that the generated textured 3D pipe model is precisely aligned in space with the actual pipe structure.

[0052] In some embodiments, ellipse fitting is performed on each closed contour to obtain the corresponding ellipse parameters, including: performing ellipse fitting on each closed contour using Hough transform to obtain the ellipse parameters corresponding to each closed contour. Alternatively, an ellipse fitting algorithm based on RANSAC is used to perform ellipse fitting on each closed contour to obtain the ellipse parameters corresponding to each closed contour.

[0053] Optionally, based on the elliptical parameters of the elliptical edge contour, a perspective transformation algorithm is used to map the elliptical edge contour in the frame image to a circular edge contour. This includes: using the two intersection points of the elliptical edge contour with the major axis and the two intersection points of the elliptical edge contour with the minor axis as source control points; constructing a target circular contour with the center of the ellipse as the center and the geometric mean of the semi-major and semi-minor axes of the ellipse as the radius; mapping the four source control points to four points on the target circular contour in the horizontal and vertical directions, corresponding to the directions of the source control points, as target control points; calculating a second perspective transformation matrix based on the correspondence between the four source control points and the four target control points; and performing a perspective transformation on the frame image using the second perspective transformation matrix to correct the elliptical edge contour in the frame image to a circular edge contour, obtaining the corrected video frame image. This completes the pipe cross-section correction of the frame image. Figure 4 As shown, Figure 4This is a schematic diagram of the corrected video frame image. Figure 4 The red lines in the text represent the rounded outline.

[0054] Optionally, based on each corrected video frame image, a panoramic image of the interior of the target pipe is obtained, including: obtaining valid annular images from each corrected video frame image; unfolding each valid annular image into a rectangular image using inverse polar coordinate transformation; and stitching the rectangular images together to obtain a panoramic image of the interior of the target pipe. The width of the panoramic image corresponds to the perimeter of the inner wall of the target pipe, and the height of the panoramic image corresponds to the length of the target pipe.

[0055] Optionally, obtaining the valid circular image in each corrected video frame image includes performing the following processing on each corrected video frame image: Based on the image resolution of the corrected video frame image, the corresponding image overlap is determined. Image overlap refers to the proportion of overlap between adjacent frames in the circumferential or axial direction of the pipeline. Based on the image overlap of the corrected video frame image, the ring width of the effective annular region in the corrected video frame image is determined. The ring width refers to the total width of the effective annular region along the radial direction. The unit of ring width is pixels. Taking the circular edge contour in the corrected video frame image as the center line, the ring width is expanded radially inwards (towards the center) and outwards (away from the center), respectively, by half. The annular region covered by this expansion is defined as the effective annular region. Pixels outside the effective annular region are set to transparent, obtaining the effective annular image in the corrected video frame image. Figure 5 As shown, Figure 5 This is a schematic diagram of an effective circular image.

[0056] Furthermore, based on the image resolution of the corrected video frame, the corresponding image overlap is determined, including: performing a lookup operation in a preset data table based on the image resolution of the corrected video frame to find the corresponding image overlap. The preset data table stores the correspondence between image resolution and image overlap.

[0057] In some embodiments, by calculation Obtain the ring width, where, For the first j The ring width corresponding to the frame-corrected video frame image. a For the first j Image overlap of video frames after frame correction L The length of the target pipe, n This represents the total number of corrected video frame images. It represents the distance between two consecutive corrected video frames.

[0058] In some embodiments, the effective annular image is a polar coordinate image with its center point as the origin. Unfolding the effective annular image into a rectangular image means converting the polar coordinate image into a rectangular image in a Cartesian coordinate system. Here, the horizontal coordinate of the rectangular image corresponds to the polar angle θ, and the vertical coordinate corresponds to the polar radius r.

[0059] Optionally, the effective annular image is unfolded into a rectangular image using inverse polar coordinate transformation, including: calculating... ,

[0060] Obtain the coordinates of the p-th pixel in the rectangular image from the valid annular image, where ( r p , θ p Let p be the polar coordinates of the p-th pixel in the valid circular image. r p Let p be the distance from the p-th pixel in the effective circular image to the center point of the effective circular image. θ p Let be the radian angle traversed by rotating clockwise from the polar axis to the p-th pixel, with a value in the range [0, 2π). The polar axis is a reference ray with 0 radians directly below the effective annular image. The minimum radius of an effective circular image, For the maximum radius of the effective circular image, ( x p , y p Let be the coordinates of the p-th pixel in the effective circular image within the rectangular image. Width of the rectangular image (column index 0 to ... ), The height of the rectangular image (row index 0 to ... ).For example, This ensures that each degree corresponds to one pixel; +1, so that each pixel radius corresponds to a row.

[0061] According to the calculated ( x p , y p The color value of the p-th pixel is assigned to the corresponding position in the rectangular image using nearest neighbor interpolation or bilinear interpolation. This process iterates through all pixels in the valid annular image to generate the complete rectangular image. For example... Figure 6 As shown, Figure 6 This is a schematic diagram of a rectangular image.

[0062] Optionally, the rectangular images are stitched together to obtain a panoramic image of the target pipe's interior. This includes placing the first rectangular image at the starting position of a pre-created blank panoramic canvas, according to the temporal order of the detected video. Starting with the second rectangular image, the remaining rectangular images are sequentially stitched onto the panoramic canvas to obtain a panoramic image of the target pipe's interior. The starting position is the bottom of the blank panoramic canvas, i.e., at coordinate 0. The bottom boundary of the panoramic image represents the starting point of the target pipe, and the top boundary represents the ending point of the target pipe. Figure 7 As shown, Figure 7 This is a schematic diagram of a panoramic image.

[0063] By uniformizing the speed of the detection video, correcting the pipe center point and pipe cross-section for each frame of the detection video, and stitching together the rectangular images corresponding to the corrected video frame images, any pixel in the resulting panoramic image can accurately represent its actual position on the target pipe.

[0064] Furthermore, the remaining rectangular images are stitched onto the panoramic canvas, including: according to the... q Determining the image overlap of the first rectangular image q The rectangular image relative to the first q - Vertical offset of 1 rectangular image , will the q The rectangular image is placed at a vertical offset from the top boundary of the previous rectangular image. The position of the pixel (i.e., the first) q The lower boundary of the image is the same as the first one. q The vertical distance between the upper boundaries of -1 image is ). For the first q In the rectangular image, with the first q -1 overlapping area of ​​rectangular images, along the vertical direction from the first... q The boundary of the rectangular image (i.e., the lower boundary of the overlapping region) to the first... q -1 The boundary of the rectangular image (i.e., the upper boundary of the overlapping area) is linearly gradiented from 100% to 0%. By using gradient transparency in the overlapping area, the image stitching transition is made natural. Where 2 ≤ q ≤ n ,and q The integer. q Image overlap characterization of rectangular images. q Zhang rectangular image and the first q- The overlapping area of ​​a rectangular image occupies the first... q The ratio of the width of the rectangular image is [0,1].

[0065] Through calculation Obtain theq The rectangular image relative to the first q - Vertical offset of 1 rectangular image , For the first q Image overlap of rectangular images, The width of the rectangular images is the same for all rectangular images.

[0066] In some embodiments, each vertex of the 3D model of the target pipe is pre-assigned UV coordinates. The U coordinate is along the circumference of the pipe, with a value range of [0,1], where U=0 and U=1 correspond to the same joint; the V coordinate is along the length of the pipe, with a value range of [0,1], where V=0 corresponds to one end of the pipe and V=1 corresponds to the other end of the pipe.

[0067] Optionally, pixels in the panoramic image inside the target pipe are mapped to the 3D model of the pipe to generate a textured 3D model of the pipe. This includes: establishing a texture mapping relationship between the panoramic image and the surface of the 3D model of the pipe. Based on the established texture mapping relationship, for any point on the surface of the 3D model of the pipe, the corresponding pixel color is obtained from the panoramic image using bilinear interpolation based on the UV coordinates, thereby rendering the panoramic image onto the surface of the 3D model of the pipe to generate a textured 3D model of the pipe.

[0068] Optionally, a texture mapping relationship is established between the panoramic image and the surface of the pipe's 3D model, including: mapping the width direction (horizontal axis) of the panoramic image to the U coordinate (circumferential direction) of the pipe's 3D model, and mapping the height direction (vertical axis) of the panoramic image to the V coordinate (length direction) of the pipe's 3D model. Wherein, the coordinates in the panoramic image are... The UV coordinates of the pixels corresponding to the surface of the pipe 3D model are: , , The width of the panoramic image. This represents the height of the panoramic image, in pixels. 0 ≤ ≤1, and For integers, 0 ≤ ≤1, and It is an integer. correspond =0 (one end of the target pipe) correspond =1 (the other end of the target pipeline). Because... 0 and For the same joint on the corresponding 3D model of the pipe, to ensure texture continuity, the texture wrapping mode is set to repeat during texture sampling, so that... The sample obtained at 1 location and 0 identical texture pixels.

[0069] In some embodiments, a three-dimensional scene is constructed on a preset display device. This scene includes geographic environment information corresponding to the target pipeline, including a geographic coordinate system, terrain data, and the actual spatial location data of features (such as tree pits, roads, and buildings). The textured 3D model of the pipeline is loaded into this scene based on its actual design coordinates or measured coordinates, thereby visually displaying the positional relationship between the target pipeline and surrounding features against a realistic geographic spatial background. This achieves high-precision and highly intuitive display of the pipeline's internal conditions within the 3D scene.

[0070] Optionally, after generating the textured 3D model of the pipe, the process further includes: acquiring defect information from the panoramic image. The defect information includes: the pixel coordinates of the defect area, the defect type, and the defect level. Using a pre-defined coordinate mapping relationship, the pixel coordinates of the defect area are calculated to obtain the spatial coordinates of the defect area on the textured 3D model of the pipe. The spatial coordinates of the defect area are then labeled with the corresponding defect information on the textured 3D model of the pipe. This allows for a more intuitive display of the pipe's defects in the 3D scene, making it easier for users to accurately locate the specific position of the defects within the pipe.

[0071] Optionally, acquiring defect information from the panoramic image includes: displaying the panoramic image on a preset display device; outlining the contours or boundaries of defect areas on the panoramic image in response to user operation commands; receiving defect information corresponding to each defect area input by the user; and associating the defect information of each defect area with the panoramic image.

[0072] The defect types include: structural defects such as cracks, deformations, misalignments, and disconnections; and functional defects such as deposits, scale, obstructions, and tree roots. Defect levels are categorized as follows: Level 1 (minor), Level 2 (moderate), Level 3 (severe), and Level 4 (major). Defect levels are determined according to industry standards.

[0073] Optionally, obtaining defect information from the panoramic image includes: inputting the panoramic image into a pre-trained defect detection neural network model, which outputs defect detection results. The defect detection results include the pixel coordinates of the defective region, the defect type, and the defect level. The defect information of each defective region is then correlated with the panoramic image.

[0074] In some embodiments, the pre-defined coordinate mapping relationship is as follows:

[0075]

[0076]

[0077] in, For the defect area on the panoramic image c pixel coordinates for Spatial coordinates on the 3D model of the pipeline These are the starting coordinates of the 3D model of the pipeline. Here are the coordinates of the endpoint of the 3D model of the pipeline. The radius of the target pipe, The width of the panoramic image. The height of the panoramic image.

[0078] In this way, displaying the textured 3D model of the pipeline in a 3D scene and marking the defect information on the textured 3D pipeline model by coordinate information can intuitively show the relationship between various defects of the target pipeline and other surrounding features, such as tree pits, roads, and buildings. This improves the visualization and positioning accuracy of defects and provides intuitive and reliable data support for further analysis of the pipeline's causes or the judgment of subsequent pipeline maintenance and treatment methods.

[0079] Combination Figure 8 As shown in the figure, this disclosure provides an apparatus 800 for three-dimensional display of pipeline inspection videos, including: a modeling module 801, an inspection video acquisition module 802, an image correction module 803, a panoramic image acquisition module 804, a model mapping module 805, and a display module 806.

[0080] Modeling module 801 is configured to acquire a 3D model of the target pipe. Detection video acquisition module 802 is configured to acquire detection video of the interior of the target pipe. Image correction module 803 is configured to correct the pipe center point and pipe cross-section for each frame of the detection video, obtaining corrected video frame images. Panoramic image acquisition module 804 is configured to acquire a panoramic image of the interior of the target pipe based on each corrected video frame image. Model mapping module 805 is configured to map pixels from the panoramic image to the 3D pipe model, generating a textured 3D pipe model. Display module 806 is configured to display the textured 3D pipe model in a preset 3D scene.

[0081] The apparatus for 3D display of pipeline inspection videos provided in this disclosure effectively eliminates image distortion caused by positional offset and perspective distortion when the camera captures images inside the pipeline by correcting the pipeline center point and pipeline cross-section of each frame of the pipeline inspection video. This results in corrected video frame images with a unified viewpoint and corrected geometric deformation. Generating panoramic images based on these corrected video frame images significantly improves the generation accuracy and geometric consistency of the panoramic images. Mapping this panoramic image to a 3D pipeline model improves the accuracy of texture mapping, ensuring that the generated textured 3D pipeline model is precisely aligned with the actual pipeline structure in space. This enhances the display accuracy of the pipeline's internal condition in a 3D scene.

[0082] Optionally, the device for 3D display of pipeline inspection videos also includes: a defect information acquisition module, a spatial coordinate acquisition module, and an annotation module.

[0083] The defect information acquisition module is configured to acquire defect information from the panoramic image, including the pixel coordinates of the defect area, the defect type, and the defect level. The spatial coordinate acquisition module is configured to calculate the pixel coordinates of the defect area using a pre-defined coordinate mapping relationship, obtaining the spatial coordinates of the defect area on the textured pipe 3D model. The annotation module is configured to annotate the spatial coordinates of the defect area on the textured pipe 3D model with the corresponding defect information.

[0084] Combination Figure 9 As shown, this disclosure provides an apparatus 900 for three-dimensional display of pipeline inspection videos, including a processor 904 and a memory 901 storing program instructions. Optionally, the apparatus may further include a communication interface 902 and a bus 903. The processor 904, communication interface 902, and memory 901 can communicate with each other via the bus 903. The communication interface 902 can be used for information transmission. The processor 904 can call the program instructions in the memory 901 to execute the method for three-dimensional display of pipeline inspection videos described in the above embodiment.

[0085] Furthermore, the logic instructions in the aforementioned memory 901 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0086] The memory 901, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 904 executes functional applications and data processing by running the program instructions / modules stored in the memory 901, thereby implementing the method for three-dimensional video display of pipeline inspection in the above embodiments.

[0087] The memory 901 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 901 may include high-speed random access memory and may also include non-volatile memory.

[0088] This disclosure provides an electronic device, including: an electronic device body, and the aforementioned device for 3D video display of pipeline inspection. The device for 3D video display of pipeline inspection is mounted on the electronic device body. The mounting relationship described herein is not limited to placement within the electronic device, but also includes mounting connections with other components of the electronic device, including but not limited to physical connections, electrical connections, or signal transmission connections. Those skilled in the art will understand that the device for 3D video display of pipeline inspection can be adapted to suitable electronic device bodies to achieve other feasible embodiments. The electronic device includes a computer or server, etc.

[0089] This disclosure provides a storage medium storing program instructions that are executed by a processor to implement the above-described method for three-dimensional video display of pipeline inspection.

[0090] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0091] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media capable of storing program code; it can also be a transient storage medium.

[0092] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0093] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0094] The methods and products disclosed in the embodiments herein (including but not limited to devices and equipment) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0095] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A method for 3D video display of pipeline inspection, characterized in that, include: Obtain the 3D model of the target pipe and the detection video of the inside of the target pipe; Each frame of the detected video is corrected for both the pipe center point and the pipe cross-section to obtain the corrected video frame image. Based on each corrected video frame image, a panoramic image of the inside of the target pipe is obtained; The pixels in the panoramic image are mapped to the 3D model of the pipeline to generate a textured 3D model of the pipeline. The textured 3D model of the pipe is displayed in a preset 3D scene.

2. The method according to claim 1, characterized in that, Pipeline center point correction is performed on each frame of the detected video, including: Perform the following processing on each frame of the detected video: The Canny edge detection operator is used to perform edge detection on the frame image to obtain several edge lines; Determine the extension direction of the target pipe in the frame image; Edge lines whose angle difference between their extension direction and the target pipe extension direction is less than a set threshold are defined as target edge lines. Calculate the coordinates of the intersection point of the extensions of any two target edge lines; Calculate the arithmetic mean of the x-coordinates and y-coordinates of all intersection points respectively to obtain the average coordinates, and determine the average coordinates as the estimated coordinates of the pipe center point; Obtain the coordinates of the center point of the frame image; Based on the center point coordinates of the frame image and the estimated coordinates of the pipe center point, the pipe center point is corrected to the center point of the frame image.

3. The method according to claim 1, characterized in that, Perform pipe cross-section correction on each frame of the detected video, including: Perform the following processing on each frame of the detected video: The Canny edge detection operator is used to perform edge detection on the frame image to obtain several edge lines; Identify the edge lines that constitute the closed contour from the aforementioned edge lines, perform ellipse fitting on each closed contour, and obtain the corresponding ellipse parameters. The closed contour edge lines corresponding to the elliptical parameters that meet the preset screening conditions are determined as the elliptical edge contours that characterize the pipe cross-section. Based on the elliptical parameters of the elliptical edge contour, the perspective transformation algorithm is used to map the elliptical edge contour into a circular edge contour.

4. The method according to claim 3, characterized in that, Based on the corrected video frame images, a panoramic image of the interior of the target pipe is obtained, including: Obtain the valid ring image from each corrected video frame; Each effective annular image is expanded into a rectangular image using inverse polar coordinate transformation; The rectangular images are stitched together to obtain a panoramic image of the inside of the target pipe.

5. The method according to claim 4, characterized in that, Obtain the valid ring image from each corrected video frame image, including: Perform the following processing on each corrected video frame: Determine the corresponding image overlap based on the image resolution of the corrected video frame; Based on the image overlap, determine the ring width of the effective annular region in the corrected video frame image; Using the circular edge contour in the corrected video frame image as the center line, extend the ring width by half to the inner and outer sides of the circle in the radial direction respectively, and determine the ring area covered by the extension as the effective ring area. Pixels outside the effective annular region are set to transparent to obtain the effective annular image in the corrected video frame image.

6. The method according to any one of claims 1 to 5, characterized in that, After generating the textured 3D model of the pipe, the following steps are also included: Obtain defect information from the panoramic image, the defect information including: pixel coordinates of the defect area, defect type, and defect level; Using a pre-defined coordinate mapping relationship, the pixel coordinates of the defect area are calculated to obtain the spatial coordinates of the defect area on the textured pipe 3D model. On the textured 3D model of the pipe, the spatial coordinates of the defect area are labeled with the corresponding defect information.

7. A device for three-dimensional video display of pipeline inspection, characterized in that, include: The modeling module is configured to acquire the 3D model of the target pipeline. The detection video acquisition module is configured to acquire detection video inside the target pipe; The image correction module is configured to perform pipe center point correction and pipe cross-section correction on each frame of the detected video to obtain corrected video frame images. The panoramic image acquisition module is configured to acquire a panoramic image of the inside of the target pipe based on each corrected video frame image; The model mapping module is configured to map pixels in the panoramic image to the 3D model of the pipeline, generating a textured 3D model of the pipeline. The display module is configured to display the textured 3D model of the pipe in a preset 3D scene.

8. An apparatus for three-dimensional video display of pipeline inspection, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to, when executing the program instructions, perform the method for three-dimensional video display of pipeline inspection as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, include: The electronic device itself; The device for three-dimensional video display of pipeline inspection as described in claim 7 or 8 is installed on the electronic device body.

10. A storage medium storing program instructions, characterized in that, The program instructions are executed by the processor to implement the method for three-dimensional video display of pipeline inspection as described in any one of claims 1 to 6.

Images