Pipeline leakage detection method, device, equipment and medium

Abstract

The invention discloses a pipeline leakage detection method, device and equipment and a medium. The method comprises the steps of determining a target pipeline image according to a target pipeline image collected by an unmanned aerial vehicle; wherein the unmanned aerial vehicle carries a visible light camera and a thermal imaging camera, and the target pipeline image comprises a target visible light image and a target thermal imaging image at the same moment; performing pipeline leakage detection on the target visible light image by using a pre-trained first detection model to obtain a first detection image; performing pipeline leakage detection on the target thermal imaging image by using a pre-trained second detection model to obtain a second detection image; and determining a pipeline leakage detection result according to the first detection image and the second detection image. According to the scheme, quick, accurate, low-cost and real-time detection of the leakage condition of the pipeline can be realized by utilizing the pipeline images shot by the visible light camera and the thermal imaging camera of the unmanned aerial vehicle, powerful technical support is provided for leakage control of an urban water supply pipe network, water resource waste is reduced, and water supply safety is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of pipeline maintenance, and particularly to a pipeline leakage detection method, device, equipment and medium. Background Art

[0002] With the acceleration of urbanization construction, the problem of water supply pipeline leakage has attracted more and more attention. Pipeline leakage may be the root cause of public health and safety problems, and the rupture of water supply pipelines is likely to lead to the entry of pollutants. Therefore, the control of water leakage is becoming increasingly important. The urban water supply industry needs leakage control measures to reduce the losses caused by leakage and generate good economic effects.

[0003] The rapid detection and accurate positioning of pipeline leakage are the prerequisites for implementing leakage control measures. Traditional leakage detection technologies are based on manual listening for leaks or simple data analysis for leak detection using various sensors. It is difficult to accurately detect tiny leakage points or leaks in hidden locations, and it is easy to miss detections. At the same time, it is easily affected by the environment, so the detection accuracy is poor. Moreover, most of the existing detection means are periodic detections or emergency detections after problems occur, so the detection real-time performance is poor. In addition, in some sparsely populated areas or areas with long pipeline layouts, the manual inspection method makes the detection cost high and the detection efficiency low. Summary of the Invention

[0004] The present invention provides a pipeline leakage detection method, device, equipment and medium to solve one of the above technical problems.

[0005] According to one aspect of the present invention, there is provided a pipeline leakage detection method, the method comprising:

[0006] Determining a target pipeline image according to a target pipeline image collected by a drone; wherein, the drone is equipped with a visible light camera and a thermal imaging camera, and the target pipeline image includes a target visible light image and a target thermal imaging image at the same moment;

[0007] Performing pipeline leakage detection on the target visible light image by using a pre-trained first detection model to obtain a first detection image;

[0008] Performing pipeline leakage detection on the target thermal imaging image by using a pre-trained second detection model to obtain a second detection image;

[0009] Determining a pipeline leakage detection result according to the first detection image and the second detection image.

[0010] According to another aspect of the present invention, there is provided a pipeline leakage detection device, the device comprising:

[0011] A target pipeline image determination module, configured to determine a target pipeline image according to a target pipeline image collected by a drone; wherein, the drone is equipped with a visible light camera and a thermal imaging camera, and the target pipeline image includes a target visible light image and a target thermal imaging image at the same moment;

[0012] A first detection image determination module, configured to perform pipeline leakage detection on the target visible light image by using a pre-trained first detection model to obtain a first detection image;

[0013] A second detection image determination module, configured to perform pipeline leakage detection on the target thermal imaging image by using a pre-trained second detection model to obtain a second detection image;

[0014] A pipeline leakage detection result determination module, configured to determine a pipeline leakage detection result according to the first detection image and the second detection image.

[0015] According to another aspect of the present invention, there is provided an electronic device, the electronic device includes:

[0016] At least one processor; and,

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the pipeline leakage detection method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, there is provided a computer-readable storage medium, the computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the pipeline leakage detection method according to any embodiment of the present invention when executed by a processor.

[0020] The technical solution of the embodiment of the present invention determines a target pipeline image according to the target pipeline image collected by the unmanned aerial vehicle (UAV); wherein, the UAV is equipped with a visible light camera and a thermal imaging camera, and the target pipeline image includes a target visible light image and a target thermal imaging image at the same moment; the first detection model trained in advance is used to detect pipeline leakage of the target visible light image to obtain a first detection image; the second detection model trained in advance is used to detect pipeline leakage of the target thermal imaging image to obtain a second detection image; the pipeline leakage detection result is determined according to the first detection image and the second detection image. In this technical solution, by combining the visible light camera and the thermal imaging camera of the UAV, using its characteristics of being flexible, having a wide field of view, being able to quickly cover a large area, and the advantage of being sensitive to temperature changes of the thermal imaging, rapid, accurate, low-cost and real-time detection of pipeline leakage is realized based on the pipeline images captured by the visible light camera and the thermal imaging camera, which provides strong technical support for the leakage control of urban water supply networks, helps to reduce water resource waste, and ensures water supply safety.

[0021] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to these drawings.

[0023] Figure 1 is a flowchart of a pipeline leakage detection method provided by the present invention;

[0024] Figure 2 is a schematic diagram of a target flight route provided by the present invention;

[0025] Figure 3 is a schematic diagram of a YOLOv11 network structure provided by the present invention;

[0026] Figure 4 is a schematic diagram of a detection image provided by the present invention;

[0027] Figure 5 is a flowchart of a pipeline leakage detection method provided by the present invention;

[0028] Figure 6 is a schematic diagram of initial image alignment provided by the present invention;

[0029] Figure 7 It is a schematic structural diagram of a pipeline leakage detection device provided according to the present invention;

[0030] Figure 8 It is a schematic structural diagram of an electronic device for implementing the pipeline leakage detection method of the present invention. Specific embodiments

[0031] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

[0032] It should be noted that the terms "first", "second", "target", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0033] Figure 1 It is a flowchart of a pipeline leakage detection method provided by the present invention. This embodiment is applicable to the situation of quickly, accurately, low-costly and real-time detecting the pipeline leakage situation. This method can be executed by a pipeline leakage detection device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device with data processing capabilities. As Figure 1 shown, the method includes:

[0034] S110, determining a target pipeline image according to the target pipeline image collected by the unmanned aerial vehicle; wherein, the unmanned aerial vehicle is equipped with a visible light camera and a thermal imaging camera, and the target pipeline image includes a target visible light image and a target thermal imaging image at the same moment.

[0035] Among them, the target pipeline may refer to the pipeline object that needs to be detected for leakage. The target pipeline image may refer to the pipeline image obtained by shooting the target pipeline with a visible light camera and a thermal imaging camera carried by a drone. Among them, the visible light camera captures the light waves of visible light in nature (wavelength range: 0.39 - 0.78 μm) for imaging, can generate color images, has RGB three channels, high image resolution, and clear details. The thermal imaging camera measures the infrared radiation emitted by the object surface (wavelength range: 0.75 - 1000 μm) for imaging, and the intensity of this radiation is related to the temperature of the object. For example, the target pipeline image may be in the form of an image or a video. The target pipeline image may refer to the pipeline image that needs to be detected for leakage. Among them, the target pipeline image may include multiple groups of target visible light images and target thermal imaging images taken at the same moment.

[0036] In this embodiment, first, use the visible light camera and the thermal imaging camera carried by the drone to shoot the target pipeline to obtain the target pipeline image, and transmit the target pipeline image to the server in real time. It should be noted that if the target pipeline image is in the form of an image, multiple pipeline images (including visible light images and thermal imaging images) taken by the drone can be directly used as the target pipeline image; if the target pipeline image is in the form of a video, multiple pipeline images (including visible light images and thermal imaging images) taken by the drone need to be generated into a corresponding video as the target pipeline image.

[0037] In this embodiment, optionally, the acquisition process of the target pipeline image includes: determining the target flight route of the drone according to the target pipeline position information, and controlling the drone to navigate according to the target flight route; during the navigation of the drone, using the visible light camera and the thermal imaging camera at the same gimbal angle to shoot the target pipeline based on a preset time interval, and obtaining multiple candidate visible light images and multiple candidate thermal imaging images respectively.

[0038] Among them, the target pipeline position information can be used to characterize the position of the target pipeline. Exemplarily, the target pipeline position information may be the longitude and latitude coordinates of the target pipeline. The target flight route may refer to the flight route planned for the drone according to the target pipeline position information. The preset time interval may refer to the camera shooting time interval preset according to actual needs. The candidate visible light image and the candidate thermal imaging image may respectively refer to the pipeline images obtained by shooting with the visible light camera and the thermal imaging camera.

[0039] Specifically, first, using the UAV positioning technology, a flight route for the UAV to fly along the target pipeline is planned according to the target pipeline position information as the target flight route of the UAV. For example, the UAV positioning technology can adopt RTK (Real-Time Kinematic) technology. Among them, RTK technology is a high-precision positioning technology applied to UAVs. Figure 2 FIG. 2 is a schematic diagram of a target flight route provided by the present invention, where 1-27 represent the sequential nodes of the UAV flight, 1 is the flight starting point, and 27 is the flight ending point.

[0040] After determining the target flight route of the UAV, control the UAV to navigate along the target flight route, and during the navigation of the UAV, use a visible light camera and a thermal imaging camera at the same gimbal angle based on a preset time interval to take pictures of the target pipeline to obtain multiple groups of candidate visible light images and candidate thermal imaging images taken at the same moment. After each shooting, the UAV transmits the candidate visible light image and candidate thermal imaging image taken at the same moment as the target pipeline image to the server in real time. Among them, shooting with the same gimbal perspective is to ensure that the visible light camera and the thermal imaging camera have the same shooting perspective. After receiving the target pipeline image, the server can directly use the candidate visible light image and candidate thermal imaging image at the same moment as the target visible light image and target thermal imaging image respectively, so as to obtain the target pipeline image.

[0041] S120, use the pre-trained first detection model to perform pipeline leakage detection on the target visible light image to obtain a first detection image.

[0042] Among them, the first detection model can refer to a machine learning model that can be used to perform pipeline leakage detection on visible light images. The first detection image can refer to the image obtained by performing pipeline leakage detection on the target visible light image using the first detection model. It can be understood that if there is a leakage in the target pipeline area corresponding to the target visible light image, the first detection image includes the pipeline leakage area. At this time, the first detection image can be understood as the target visible light image identifying the pipeline leakage area; if there is no leakage in the target pipeline area corresponding to the target visible light image, the first detection image does not include the pipeline leakage area, and at this time, the first detection image is the same as the target visible light image.

[0043] Exemplarily, the first detection model can be obtained through model training based on the YOLOv11 deep learning network structure using a supervised training method. Figure 3 FIG. 3 is a schematic diagram of the YOLOv11 network structure provided by the present invention. As Figure 3As shown, the YOLOv11 network structure consists of three main parts: the Backbone, the Neck, and the Head. Specifically, the main task of the Backbone is to extract features from the input image. In YOLOv11, the Backbone is composed of multiple modules, including the Convolutional Block Stem (CBS) and the Convolutional Block with 3x3 and 2x2 kernels (C3K2). Among them, the Convolutional Block contains one or more convolutional layers for initial feature extraction, that is, converting the original image into a higher-level feature representation; the Convolutional Block with 3x3 and 2x2 kernels contains 3×3 and 2×2 convolutional kernels for further feature extraction and increasing the depth of the network. The input image passes through a series of CBS and C3K2 modules, gradually reducing the spatial resolution (i.e., the image size) while increasing the number of feature channels (i.e., the number of feature maps). For example, the size of the feature map gradually decreases from 640×640 to 256×20×20, while the number of feature channels increases from 3 to a larger value.

[0044] The main task of the Neck is to fuse features at different levels and perform feature enhancement, which is achieved through upsampling, downsampling, and feature fusion. Among them, Upsample represents the upsampling operation, which is used to increase the spatial resolution of the feature map and align it with the higher-resolution feature map; Contact represents the feature fusion operation, usually by concatenating feature maps of different resolutions together. Specifically, the feature map gradually recovers to a higher resolution through multiple upsampling and contact operations, and different-level features are continuously fused during this process. Finally, multi-scale feature maps are obtained, which contain information at different levels and are helpful for detecting targets of different sizes.

[0045] The main task of the Head is to make the final predictions based on the fused feature maps, including the position and category of the bounding box. Among them, DSC represents the depthwise separable convolution, which is used to reduce the computational cost while maintaining the model performance; Conv2d represents the final convolutional layer, which is used to generate predictions for the bounding box and category. The fused feature map is processed through multiple CBS and DSC modules, and finally, the prediction results are generated through the Conv2d layer. Each output branch corresponds to feature maps of different scales, respectively responsible for detecting targets of different sizes.

[0046] In this embodiment, after determining the target pipeline image, the target visible light image in the target pipeline image can be input into a pre-trained first detection model. The first detection model is used to perform inference segmentation on the target visible light image, and the mask regions of ground water accumulation and water seepage in the target visible light image are extracted as the pipeline leakage regions, so as to realize the pipeline leakage detection of the target visible light image. After the pipeline leakage detection is completed, the first detection model will output the first detection image corresponding to the target visible light image.

[0047] S130. Use a pre-trained second detection model to perform pipeline leakage detection on the target thermal imaging image to obtain a second detection image.

[0048] Among them, the second detection model can refer to a machine learning model that can be used for pipeline leakage detection of thermal imaging images. Exemplarily, the second detection model can be obtained through model training based on the YOLOv11 deep learning network structure using a supervised training method. The second detection image can refer to the image obtained by using the second detection model to perform pipeline leakage detection on the target thermal imaging image. It can be understood that if there is a leakage in the target pipeline area corresponding to the target thermal imaging image, the second detection image includes the pipeline leakage area. At this time, the second detection image can be understood as the target thermal imaging image identifying the pipeline leakage area; if there is no leakage in the target pipeline area corresponding to the target thermal imaging image, the second detection image does not include the pipeline leakage area. At this time, the second detection image is the same as the target thermal imaging image.

[0049] In this embodiment, after determining the target pipeline image, the target thermal imaging image in the target pipeline image can be input into a pre-trained second detection model. The second detection model is used to perform inference segmentation on the target thermal imaging image, and the mask regions of ground water accumulation and water seepage in the target thermal imaging image are extracted as the pipeline leakage regions, so as to realize the pipeline leakage detection of the target thermal imaging image. After the pipeline leakage detection is completed, the second detection model will output the second detection image corresponding to the target thermal imaging image. Figure 4 It is a schematic diagram of a detection image provided by the present invention. Among them, the left figure is the second detection image corresponding to the target thermal imaging image, and the right figure is the first detection image corresponding to the target visible light image. The black cloud-like regions in the left and right figures are the detected water leakage regions (i.e., pipeline leakage regions).

[0050] S140. Determine the pipeline leakage detection result according to the first detection image and the second detection image.

[0051] In this embodiment, after obtaining the first detection image and the second detection image, the pipeline leakage detection result can be determined based on the first detection image and the second detection image. Among them, the pipeline leakage detection result may include whether there is leakage in the target pipeline and the warning level of the leakage. Exemplarily, the warning level can be characterized in the form of a level or a score. It can be understood that the higher the warning level, the greater the possibility that there is leakage in the target pipeline. After determining the pipeline leakage detection result, a warning can be issued for the situation where there is pipeline leakage in the pipeline leakage detection result. For example, the warning method can adopt voice broadcast or reminder by an audible and visual alarm lamp, etc. Among them, different warning levels can be set with different intensities (such as sound intensity) or frequencies of warning prompts.

[0052] In this embodiment, optionally, determining the pipeline leakage detection result based on the first detection image and the second detection image includes: if there are leakage regions in both the first detection image and the second detection image, determining whether there is an overlap between the leakage regions in the first detection image and the second detection image; if there is an overlap, determining that there is leakage in the target pipeline and the warning level is the highest level.

[0053] Specifically, if there are leakage regions in both the first detection image and the second detection image, it is necessary to further determine whether there is an overlap between the leakage regions in the first detection image and the second detection image. When there is an overlap between the leakage regions in the first detection image and the second detection image, it can be directly determined that there is leakage in the target pipeline and the corresponding warning level is the highest level.

[0054] In this embodiment, optionally, determining the pipeline leakage detection result based on the first detection image and the second detection image includes: if there is a leakage region in the first detection image and there is no leakage region in the second detection image, determining, according to the image mapping relationship between the first detection image and the second detection image, the image region corresponding to the leakage region of the first detection image in the second detection image as the reference leakage region; taking the image region within a preset range around the reference leakage region as the background region of the reference leakage region, and determining the first color feature information of the reference leakage region and the second color feature information of the background region; if the difference between the first color feature information and the second color feature information is greater than a preset threshold, determining that there is leakage in the target pipeline and the warning level is the intermediate level.

[0055] Among them, the image mapping relationship can be used to describe the transformation relationship between images. For example, the transformation relationship can include affine transformation and perspective transformation, etc. The color feature information can be used to describe the regional temperature feature and can be specifically determined based on the color histogram. Exemplarily, the color feature information can include at least one of the color average value, the color maximum value, and the color minimum value.

[0056] Specifically, if there is a leakage area in the first detection image and no leakage area in the second detection image, it is necessary to determine the image area corresponding to the leakage area of the first detection image in the second detection image as the reference leakage area according to the image mapping relationship between the first detection image and the second detection image, and use the image area within a preset range around the reference leakage area as the background area of the reference leakage area. Among them, in this embodiment, the size and shape of the preset range are not specifically limited and can be determined according to actual needs. Exemplarily, one or more image areas can be selected around the reference leakage area as the background area, and the selected image areas can be circular, elliptical, rectangular or square, etc., and the sizes of multiple image areas can be the same or different. In addition, the other areas except the reference leakage area in the image area determined by the minimum bounding box of the reference leakage area can also be used as the background area, where the minimum bounding box can be circular, elliptical, rectangular or square, etc.

[0057] After determining the reference leakage area and its background area, the first color feature information of the reference leakage area and the second color feature information of the background area can be determined respectively. Exemplarily, the pixel color histograms of the reference leakage area and its background area can be determined respectively, and the first color feature information of the reference leakage area and the second color feature information of the background area can be calculated according to the pixel color histograms, and then the difference between the first color feature information and the second color feature information can be calculated. If the difference between the first color feature information and the second color feature information is greater than the preset threshold, it can be determined that the target pipeline has a leakage and the corresponding warning level is the intermediate level. It should be noted that if the color feature information includes multiple parameters, it is necessary to calculate the differences between the first color feature information and the second color feature information corresponding to each parameter respectively, and compare the differences corresponding to each parameter with the preset threshold corresponding to the parameter. Therefore, it is necessary to preset the corresponding preset thresholds for different parameters in advance.

[0058] In this embodiment, optionally, determining the pipeline leakage detection result according to the first detection image and the second detection image includes: if there is a leakage area in the second detection image and no leakage area in the first detection image, reducing the confidence of the first detection model to the reference threshold to obtain an updated first detection model; using the updated first detection model to perform pipeline leakage detection on the target visible light image to obtain a reference detection image; if there is a leakage area in the reference detection image and there is an overlap with the leakage area in the second detection image, it is determined that the target pipeline has a leakage and the warning level is the lowest level.

[0059] Specifically, if there is a leakage area in the second detection image and no leakage area in the first detection image, the confidence level of the first detection model can be reduced to a reference threshold to obtain an updated first detection model, and then the updated first detection model is used to detect pipeline leakage in the target visible light image to obtain a reference detection image. Among them, the confidence level is related to the model accuracy, and the reference threshold can be preset according to the actual model accuracy requirements. If there is no leakage area in the reference detection image, it can be determined that the target pipeline has no leakage, and no warning is required at this time; if there is a leakage area in the reference detection image and it overlaps with the leakage area in the second detection image, it can be determined that the target pipeline has a leakage and the corresponding warning level is the lowest level.

[0060] Furthermore, if there are leakage areas in both the first detection image and the second detection image and the leakage areas in the first detection image and the second detection image do not overlap, the leakage area in the first detection image can be analyzed by referring to the situation where there is a leakage area in the first detection image and no leakage area in the second detection image, and at the same time, the leakage area in the second detection image can be analyzed by referring to the situation where there is a leakage area in the second detection image and no leakage area in the first detection image, so as to judge whether the target pipeline has a leakage and the warning level when there is a leakage.

[0061] The technical solution of the embodiment of the present invention determines a target pipeline image according to the target pipeline image collected by the unmanned aerial vehicle; wherein, the unmanned aerial vehicle is equipped with a visible light camera and a thermal imaging camera, and the target pipeline image includes a target visible light image and a target thermal imaging image at the same time; the first detection model trained in advance is used to detect pipeline leakage in the target visible light image to obtain a first detection image; the second detection model trained in advance is used to detect pipeline leakage in the target thermal imaging image to obtain a second detection image; the pipeline leakage detection result is determined according to the first detection image and the second detection image. This technical solution combines the visible light camera and the thermal imaging camera of the unmanned aerial vehicle, utilizes its characteristics of being flexible, having a wide field of view, being able to quickly cover a large area, and the advantage of being sensitive to temperature changes of the thermal imaging, and realizes the rapid, accurate, low-cost and real-time detection of pipeline leakage based on the pipeline images taken by the visible light camera and the thermal imaging camera, provides strong technical support for the leakage control of urban water supply networks, helps to reduce water resource waste, and ensures water supply safety.

[0062] In this embodiment, optionally, the process of collecting the target pipeline image includes: determining the target flight route of the drone according to the target pipeline position information, and controlling the drone to navigate according to the target flight route; during the navigation of the drone, using a visible light camera and a thermal imaging camera to take pictures of the target pipeline at the same gimbal angle, respectively obtaining a plurality of candidate visible light images and a plurality of candidate thermal imaging images; performing image stitching on the candidate visible light images and candidate thermal imaging images at each same moment based on a preset stitching method to obtain a plurality of candidate stitched images; generating a target pipeline video according to the plurality of candidate stitched images in chronological order.

[0063] Among them, the preset stitching method may refer to an image stitching method preset according to actual needs. Exemplarily, the preset stitching method may include an image stitching size and an image stitching order. Among them, the image stitching size may be determined based on the size of the candidate visible light image and / or the candidate thermal imaging image. For example, the smaller image size or the larger image size among the candidate visible light image and the candidate thermal imaging image may be used as the image stitching size, or the average image size of the candidate visible light image and the candidate thermal imaging image may be used as the image stitching size. In particular, if the candidate visible light image and the candidate thermal imaging image have the same image size, the size of the candidate visible light image or the candidate thermal imaging image may be directly used as the image stitching size. Among them, the image stitching order may adopt the order of the candidate visible light image first and the candidate thermal imaging image second, or the order of the candidate thermal imaging image first and the candidate visible light image second.

[0064] In this embodiment, when collecting the target pipeline image, first, the target flight route of the UAV is determined according to the target pipeline position information (such as longitude and latitude coordinates) by using the UAV positioning technology. Then, the UAV is controlled to navigate according to the target flight route. During the navigation of the UAV, a visible light camera and a thermal imaging camera are used to take real-time pictures of the target pipeline at the same pan-tilt angle, respectively obtaining a plurality of candidate visible light images and a plurality of candidate thermal imaging images. Furthermore, based on a preset stitching method, the candidate visible light images and candidate thermal imaging images at each same moment are stitched to obtain a plurality of candidate stitched images. It should be noted that when the candidate visible light images and candidate thermal imaging images have the same image size, the candidate visible light images and candidate thermal imaging images at each same moment can be directly stitched according to the image stitching order in the preset stitching method to obtain candidate stitched images at multiple moments; when the candidate visible light images and candidate thermal imaging images have different image sizes, the candidate visible light images and / or candidate thermal imaging images need to be scaled according to the image stitching size in the preset stitching method so that both sizes are equal to the image stitching size, and then the candidate visible light images and candidate thermal imaging images that have undergone image scaling at each same moment are stitched according to the image stitching order in the preset stitching method to obtain candidate stitched images at multiple moments. After obtaining the candidate stitched images at multiple moments, a corresponding video can be generated as the target pipeline video according to the multiple candidate stitched images in chronological order. After determining the target pipeline video, the generated target pipeline video can be transmitted to the server by the UAV.

[0065] With such a setting, the candidate visible light images and candidate thermal imaging images at the same moment are stitched, which helps to align the time of the two images; generating a target pipeline video based on the multiple candidate stitched images after image stitching helps to improve the transmission efficiency of video data compared with generating a visible light video and a thermal imaging video separately.

[0066] In this embodiment, optionally, determining the target pipeline image according to the target pipeline image collected by the UAV includes: determining the target frame extraction interval of the target pipeline video based on a preset frame extraction formula; performing frame extraction processing on the target pipeline video according to the target frame extraction interval to obtain a plurality of target stitched images; and performing image segmentation on each target stitched image to obtain a plurality of target visible light images and target thermal imaging images at the same moment; where the expression of the preset frame extraction formula is as follows:

[0067]

[0068] where N represents the frame extraction interval, T represents the pipeline leakage detection time, t represents the maximum detection time, and respectively represent the minimum frame extraction interval and the maximum frame extraction interval.

[0069] It should be noted that since there are a large number of candidate stitching images included in the target pipeline video, if the visible light images and thermal imaging images in each candidate stitching image are used for pipeline leakage detection, it will cost a considerable amount of computing cost and time cost, resulting in poor real-time performance of pipeline leakage detection. Therefore, it is necessary to perform frame extraction on the target pipeline video and select a part of the candidate stitching images in the target pipeline video to participate in pipeline leakage detection.

[0070] In order to ensure the real-time performance of pipeline leakage detection and the smoothness of the streaming video, the present invention designs a dynamic frame extraction detection method to perform frame extraction on the target pipeline video. Specifically, first, use the preset frame extraction formula to determine the frame extraction interval corresponding to the target pipeline video as the target frame extraction interval. Wherein, N represents the frame extraction interval, T represents the pipeline leakage detection time, t represents the maximum detection time, and respectively represent the minimum frame extraction interval and the maximum frame extraction interval. Wherein, T can be expressed as the total time of image preprocessing (such as pixel normalization), model inference and post-processing (such as optimizing the extracted mask area); t can be set based on the requirement of video smoothness; setting and is to prevent the unreasonable number of extracted frames caused by abnormal inference time such as the warm-up of the model during the first inference.

[0071] After determining the target frame extraction interval, multiple target stitching images can be obtained by performing frame extraction on the target pipeline video according to the target frame extraction interval, and then image segmentation is performed on each target stitching image to obtain multiple target visible light images and target thermal imaging images at the same moment. Then, the target visible light image is input into the first detection model for pipeline defect detection to obtain a first detection image, and at the same time, the target thermal imaging image is input into the second detection model for pipeline defect detection to obtain a second detection image. Finally, the final pipeline leakage detection result is determined according to the first detection image and the second detection model.

[0072] With such a setting in this solution, the dynamic frame extraction detection method can be used to perform frame extraction on the target pipeline video, thereby ensuring the real-time performance of pipeline leakage detection and the smoothness of the streaming video.

[0073] In this embodiment, optionally, after determining the pipeline leakage detection result based on the first detection image and the second detection image, the method further includes: determining a target pipeline image with pipeline leakage in the target pipeline as a pipeline leakage image according to the pipeline leakage detection result, and obtaining the longitude and latitude information of the target drone when the pipeline leakage image is captured; determining one of the multiple preset fire hydrants deployed around the target pipeline as the target fire hydrant, and obtaining the target identification information of the target fire hydrant; wherein, the difference between the longitude and latitude information of the target fire hydrant and the longitude and latitude information of the target drone is the smallest; determining the relative angle and relative distance between the longitude and latitude information of the target drone and the longitude and latitude information of the target fire hydrant, and positioning the pipeline leakage location according to the target identification information and the relative angle and relative distance.

[0074] Wherein, the pipeline leakage image may refer to the target pipeline image with a pipeline leakage area. The longitude and latitude information of the target drone can be used to represent the position of the drone when the pipeline leakage image is captured. It should be noted that multiple fire hydrants are pre-deployed around the target pipeline for providing fire-fighting water, and each fire hydrant has a unique identification information. For example, the identification information may be a number. The target fire hydrant may refer to the preset fire hydrant corresponding to the longitude and latitude information with the smallest difference from the longitude and latitude information of the target drone. The target identification information can be used to uniquely identify the target fire hydrant.

[0075] It should be noted that since the drone flies along the target pipeline and the height difference is not considered, the longitude and latitude information of the drone can be approximately regarded as the longitude and latitude information of the target pipeline. However, for the management personnel, it is difficult to quickly locate the pipeline leakage location only with the longitude and latitude information of the target pipeline. To solve the above problems, in this embodiment, the fire hydrants pre-deployed around the target pipeline are used to achieve the quick positioning of the pipeline leakage location.

[0076] Specifically, after determining the pipeline leakage detection result based on the first detection image and the second detection image, first, select the target pipeline image with leakage in the target pipeline as the pipeline leakage image according to the pipeline leakage detection result, and then obtain the longitude and latitude information of the drone when the pipeline leakage image is taken as the target drone longitude and latitude information. Among them, the target drone longitude and latitude information can be used to approximately represent the leakage location of the target pipeline. Furthermore, determine the longitude and latitude information of a fire hydrant with the smallest difference from the target drone longitude and latitude information based on the longitude and latitude information of multiple preset fire hydrants deployed around the target pipeline, and use the corresponding preset fire hydrant as the target fire hydrant, and then obtain the target identification information of the target fire hydrant. Exemplarily, the longitude and latitude information of the preset fire hydrant can be used to construct a KD tree in advance to utilize the KD tree for efficient spatial search, so as to quickly find the longitude and latitude information of the fire hydrant with the smallest difference from the target drone longitude and latitude information. Then use the Haversine formula to calculate the relative angle between the target drone longitude and latitude information and the longitude and latitude information of the target fire hydrant, and use the geodetic library to calculate the relative distance between the target drone longitude and latitude information and the longitude and latitude information of the target fire hydrant. Finally, based on the target identification information, relative angle and relative distance, the rapid positioning of the pipeline leakage location can be realized.

[0077] Furthermore, the warning level and the target fire hydrant information (including the target identification information, relative angle and relative distance) can also be added to the detection images with pipeline leakage (including the first detection image and / or the second detection image) to obtain the target detection image, and the target detection image is uploaded to the warning platform as the pipeline leakage detection result for warning according to the pipeline leakage detection result.

[0078] With such a setting in this solution, the rapid positioning of the leakage location of the target pipeline is realized by means of the fire hydrants pre-deployed around the target pipeline, so that the management personnel can quickly find the leakage location of the target pipeline for subsequent maintenance and treatment.

[0079] Figure 5 It is a flowchart of a pipeline leakage detection method provided by the present invention, and this embodiment is optimized based on the above embodiment. The specific optimization is as follows: Before determining the pipeline leakage detection result based on the first detection image and the second detection image, it further includes: performing initial image alignment on the first detection image and the second detection image according to the camera parameter information and shooting attitude information of the drone to obtain the first alignment result; performing secondary image alignment on the first alignment result based on the feature matching algorithm to obtain the second alignment result; updating the images of the first detection image and the second detection image according to the second alignment result.

[0080] As Figure 5 shown, the method of this embodiment specifically includes the following steps:

[0081] S210. Determine the target pipeline image based on the target pipeline images collected by the drone. Among them, the drone is equipped with a visible light camera and a thermal imaging camera, and the target pipeline image includes the target visible light image and the target thermal imaging image at the same moment.

[0082] S220. Use the pre-trained first detection model to perform pipeline leakage detection on the target visible light image to obtain the first detection image.

[0083] S230. Use the pre-trained second detection model to perform pipeline leakage detection on the target thermal imaging image to obtain the second detection image.

[0084] Among them, the specific implementation manners of S210 - S230 can refer to the above relevant descriptions and will not be elaborated here.

[0085] S240. Perform initial image alignment on the first detection image and the second detection image according to the camera parameter information and the shooting attitude information of the drone to obtain the first alignment result.

[0086] Among them, the camera parameter information may include the camera sensor size, the camera focal length, the camera zoom ratio, and the camera resolution, etc. The shooting attitude information may include the shooting height of the drone. The first alignment result may refer to the image result obtained after performing initial image alignment on the first detection image and the second detection image.

[0087] It should be noted that since the parameters of the visible light camera and the thermal imaging camera are quite different, and both may use zoom shooting during the flight of the drone, there are uncontrollable deviations in the content of the two images at the same time node. As Figure 4 shown, the black leakage area is the same target, but there are deviations in the position and size in the two images. For this reason, the present invention proposes an efficient and fast mapping method (i.e., the fast alignment method for dual - light images), which can perform initial image alignment on the first detection image and the second detection image according to the camera parameter information and the shooting attitude information of the drone. Among them, the fast alignment method for dual - light images does not require obtaining the camera internal parameters and external parameters through complex and cumbersome camera calibration pre - work, and can also adapt to the real - time zoom working conditions of the zoom lens during operation.

[0088] In this embodiment, optionally, performing initial image alignment on the first detection image and the second detection image according to the camera parameter information and the shooting attitude information of the drone to obtain the first alignment result includes the following steps A1 - A6:

[0089] A1. Determine the shooting angle of the visible light camera according to the camera parameter information of the visible light camera, and determine the shooting angle of the thermal imaging camera according to the camera parameter information of the thermal imaging camera.

[0090] Among them, the shooting perspectives can include a horizontal perspective and a vertical perspective. The horizontal perspective can be used to represent the shooting perspective of the camera in the horizontal direction, and the vertical perspective can be used to represent the shooting perspective of the camera in the vertical direction. Exemplarily, assuming that the camera sensor size is m×n, the camera focal length is f, the camera zoom ratio is Z (if the camera does not zoom, Z is defaulted to 1), and the camera resolution is M×N. The horizontal perspective of the camera can be calculated by the first formula α m = 2×arctan[m / (2×f×Z)], and the vertical perspective of the camera can be calculated by the second formula α n = 2×arctan[n / (2×f×Z)]. Thus, by substituting the camera parameter information of the visible light camera and the thermal imaging camera into the first formula and the second formula respectively, the horizontal perspective and the vertical perspective of the visible light camera can be obtained as α m1 and α n1 , and the horizontal perspective and the vertical perspective of the thermal imaging camera can be obtained as α m2 and α n2 .

[0091] A2. Determine the shooting field of view of the visible light camera according to the shooting perspective of the visible light camera and the shooting height of the drone, and determine the shooting field of view of the thermal imaging camera according to the shooting perspective of the thermal imaging camera and the shooting height of the drone.

[0092] Among them, the shooting field of view can include a horizontal shooting field of view and a vertical shooting field of view. The horizontal shooting field of view can be used to represent the maximum distance of the image captured by the camera in the horizontal direction of the physical world, and the vertical shooting field of view can be used to represent the maximum distance of the image captured by the camera in the vertical direction of the physical world. Exemplarily, assuming that the shooting height of the drone is h, the horizontal field of view of the camera can be calculated by the third formula L m = tan(α m / 2)×h, and the vertical field of view of the camera can be calculated by the fourth formula L n = tan(α n / 2)×h. Thus, by substituting the shooting perspectives and the shooting height of the drone of the visible light camera and the thermal imaging camera into the first formula and the second formula respectively, the horizontal shooting field of view and the vertical shooting field of view of the visible light camera can be obtained as L m1 and L n1 , and the horizontal shooting field of view and the vertical shooting field of view of the thermal imaging camera can be obtained as L m2 and L n2 .

[0093] A3. Determine the source image and the target image from the first detection image and the second detection image according to the shooting field of view of the visible light camera and the shooting field of view of the thermal imaging camera; among them, the shooting field of view of the source image is smaller than that of the target image.

[0094] Specifically, by comparing the shooting fields of view of the visible light camera and the thermal imaging camera, the minimum shooting field of view and the maximum shooting field of view can be determined. In the first detection image and the second detection image, the detection image corresponding to the minimum shooting field of view is determined as the source image, and the detection image corresponding to the maximum shooting field of view is determined as the target image to determine the source image and the target image. Exemplarily, taking Figure 4 as an example, by calculating the shooting fields of view of the two detection images (including the horizontal shooting field of view and the vertical shooting field of view), it can be known that L m2 >L m1 and L n2 >L n1 , that is, the shooting field of view of the second detection image is larger than that of the first detection image. At this time, the first detection image can be determined as the source image, and the second detection image can be determined as the target image.

[0095] A4. Determine the pixel field of view of the target image according to the shooting field of view of the target image and the camera resolution corresponding to the target image; wherein, the pixel field of view is used to characterize the field of view size occupied by each pixel point in the physical world in the image.

[0096] Among them, the pixel field of view can include a horizontal pixel field of view and a vertical pixel field of view. The horizontal pixel field of view can be used to characterize the field of view size occupied by the horizontal distance of each pixel point in the physical world in the image, and the vertical pixel can be used to characterize the field of view size occupied by the vertical distance of each pixel point in the physical world in the field of view image. Exemplarily, the horizontal pixel field of view can be calculated by the fifth formula P m =L m / M, and the vertical pixel field of view can be calculated by the sixth formula P n =L n / N. Substitute the shooting field of view of the target image (including the horizontal shooting field of view and the vertical shooting field of view) and the camera resolution corresponding to the target image into the fifth formula and the sixth formula respectively, and the pixel field of view of the target image (including the horizontal pixel field of view and the vertical pixel field of view) can be obtained. For example, assuming that the second detection image is determined as the target image, the horizontal pixel field of view and the vertical pixel field of view of the target image can be calculated as P m2 and P n2 respectively through the fifth formula and the sixth formula.

[0097] A5. Determine the target size according to the shooting field of view of the source image and the pixel field of view of the target image, and crop the target image according to the target size to obtain the first image.

[0098] Among them, the target size may refer to the image size after the initial image alignment, specifically including the target horizontal size and the target vertical size. The first image may refer to a new image obtained by cropping the target image according to the target size. It should be noted that since the positions of the two cameras are close to each other during the drone shooting and the shooting postures are nearly the same, assuming that the central point area of the visible light and thermal imaging images is the same position, only the pixel fields of view of the visible light and thermal imaging captured images need to be calculated to determine the scaling ratio between the visible light image and the thermal imaging image. It can be understood that since the shooting field of view of the source image is smaller than that of the target image, in order to perform image alignment, it is necessary to crop the target image with a larger shooting field of view.

[0099] Exemplarily, assuming that the first detection image is the source image and the second detection image is the target image, the target horizontal size can be calculated by the seventh formula m c = L m1 / P m2 and the target vertical size can be calculated by the eighth formula n c = L n1 / P n2 Then, taking the center point of the second detection image as a reference, crop a new image with a pixel size of m c *n c that is mapped to align with the first detection image as the first image, as Figure 6 shown.

[0100] A6. Adjust the image size of the first image according to the image size of the source image to obtain a second image, and determine the first alignment result based on the source image and the second image.

[0101] After obtaining the first image, the image size of the first image can be scaled according to the image size of the source image to obtain a second image, and the second image is made to have the same image size as the source image through image size adjustment. Thus, the source image and the second image can be used as the first alignment result.

[0102] With such a setting in this solution, the initial image alignment of the first detection image and the second detection image can be performed using the dual - light image fast alignment method, effectively overcoming the problem that there is a deviation in the picture content between the visible light image and the thermal imaging image at the same time node due to the large difference between the parameters of the visible light camera and the thermal imaging camera, which helps to improve the accuracy of pipeline leakage detection.

[0103] S250. Based on the feature matching algorithm, perform secondary image alignment on the first alignment result to obtain a second alignment result.

[0104] It should be noted that due to reasons such as camera lens distortion and parameter errors during operation and shooting, there will be a certain error in the mapping between the source image and the second image. Therefore, the present invention uses a feature matching algorithm to further align the source image and the second image, which helps to further improve the accuracy of image alignment.

[0105] In this embodiment, optionally, performing a second image alignment on the first alignment result based on the feature matching algorithm includes: using the feature matching algorithm to determine the feature point matching pairs between the source image and the second image; determining the image mapping relationship between the source image and the second image according to the feature point matching pairs; correcting the second image according to the image mapping relationship to obtain a third image, and determining the second alignment result according to the source image and the third image.

[0106] Specifically, first, use the feature matching algorithm to extract image feature points from the source image and the second image, and match the feature points in the two images to obtain feature point matching pairs. Exemplarily, the ORB (Oriented FAST and Rotated BRIEF) algorithm can be used for feature point extraction. Among them, the ORB algorithm is a feature detection algorithm that combines FAST corner detection and BRIEF descriptor and performs rotation invariance processing on them, and can effectively extract key feature points in the image. When performing feature point matching, the present invention uses the image feature points and their descriptors extracted by the ORB algorithm, and pairs them through brute force matching (Brute Force Matcher) to calculate the similarity of each feature point in the image. To improve the matching accuracy, the ratio test (Lowe's ratio test) is used to eliminate incorrect matches by comparing the distance ratio between the best matching point and the second-best matching point. After optimization and verification, the above ORB algorithm can have good adaptability to targets with certain changes in shape, color, and size.

[0107] After determining the feature point matching pairs between the source image and the second image, the geometric transformation parameters can be calculated through these feature point matching pairs, and the geometric transformation parameters are determined as the image mapping relationship between the source image and the second image. Exemplarily, the geometric transformation parameters can include an affine transformation matrix or a homography matrix, etc. After determining the image mapping relationship, the image mapping relationship can be applied to the second image for correction to obtain a third image, and the source image and the third image are used as the second alignment result. When applying the image mapping relationship to the second image correction, each pixel point in the second image can be directly transformed using the affine transformation matrix, and then the pixel value can be calculated using bilinear interpolation; or the pixel position of the source image can be calculated through reverse transformation using the homography matrix, and then the pixel value in the second image can be filled through interpolation.

[0108] With such a setting, this solution uses the feature matching alignment method to perform image alignment on the first alignment result again, effectively overcoming the problem of certain errors in image mapping caused by reasons such as camera lens distortion and shooting parameter errors, which helps to improve the accuracy of pipeline leakage detection.

[0109] S260. Update the first detection image and the second detection image according to the second alignment result.

[0110] In this embodiment, after obtaining the second alignment result, the detection images of the same image type can be updated according to the second alignment result. Among them, the image types include visible light images and thermal imaging images. Exemplarily, assuming that the image type of the source image is a visible light image and the image type of the third image is a thermal imaging image, the first detection image needs to be updated to the source image, and the second detection image needs to be updated to the third image, so as to realize the update of the first detection image and the second detection image.

[0111] S270. Determine the pipeline leakage detection result according to the updated first detection image and the second detection image.

[0112] In the technical solution of the embodiment of the present invention, before determining the pipeline leakage detection result according to the first detection image and the second detection image, perform initial image alignment on the first detection image and the second detection image according to the camera parameter information and shooting attitude information of the drone to obtain the first alignment result; perform secondary image alignment on the first alignment result based on the feature matching algorithm to obtain the second alignment result; update the first detection image and the second detection image according to the second alignment result. This technical solution effectively overcomes the problem that the picture content of the visible light image and the thermal imaging image at the same time node is deviated due to the large difference between the visible light camera parameters and the thermal imaging camera parameters through the initial image alignment, effectively overcomes the problem of certain errors in image mapping caused by reasons such as camera lens distortion and shooting parameter errors through the secondary image alignment, and helps to further improve the accuracy of pipeline leakage detection through the two image alignment processes.

[0113] In this embodiment, optionally, after determining the feature point matching pairs between the source image and the second image by using the feature matching algorithm, the following steps are further included: determining the first feature point contour information according to the source image feature points in the feature point matching pairs, and determining the second feature point contour information according to the second image feature points in the feature point matching pairs; determining the contour similarity between the first feature point contour information and the second feature point contour information, and determining the target transformation model according to the contour similarity; wherein, the transformation model includes an affine transformation model and a perspective transformation model; correspondingly, determining the image mapping relationship between the source image and the second image according to the feature point matching pairs includes: determining the geometric transformation parameters between the source image and the second image according to the target transformation model and the feature point matching pairs; wherein, the geometric transformation parameters are an affine transformation matrix or a homography matrix; and determining the geometric transformation parameters as the image mapping relationship between the source image and the second image.

[0114] Wherein, the first feature point contour information and the second feature point contour information can be respectively used to characterize the feature point contours in the source image and the second image involved in the feature point matching pairs. The contour similarity can be used to characterize the similarity degree between the first feature point contour information and the second feature point contour information. The transformation model can be used to determine the image mapping relationship.

[0115] In this embodiment, the complexity of the image change can be judged according to the feature point contour similarity, and different transformation models can be used to determine the image mapping relationship based on the complexity of the image change. Specifically, first, determine the feature point contour information corresponding to the source image feature points in the feature point matching pairs as the first feature point contour information, and determine the feature point contour information corresponding to the second image feature points in the feature point matching pairs as the second feature point contour information. Then calculate the contour similarity between the first feature point contour information and the second feature point contour information, and determine the target transformation model according to the contour similarity. Exemplarily, the contour similarity can be calculated based on methods such as the Hausdorff distance, the Fréchet distance, the shape context, or the Fourier descriptor.

[0116] Among them, the transformation model includes an affine transformation model and a perspective transformation model. Specifically, the affine transformation model is applicable to simple image transformations such as translation, rotation, and scaling, and can be used to solve the affine transformation matrix; the perspective transformation model is applicable to complex image transformations with perspective distortion and can be used to solve the homography matrix. Optionally, determining the target transformation model according to the contour similarity includes: if the contour similarity is greater than the preset similarity, determining the target transformation model as the affine transformation model; if the contour similarity is less than or equal to the preset similarity, determining the target transformation model as the perspective transformation model. Among them, the preset similarity can be a reference value of the contour similarity preset according to actual needs. It can be understood that the higher the contour similarity, the simpler the image transformation. At this time, the affine transformation model can be used to solve the affine transformation matrix as the image mapping relationship between the source image and the second image; the lower the contour similarity, the more complex the image transformation. At this time, the perspective transformation model can be used to solve the homography matrix as the image mapping relationship between the source image and the second image. Among them, the affine transformation matrix includes rotation, scaling, and translation parameters, which can be solved by the least squares method. Exemplarily, the homography matrix can be 3×3 in size.

[0117] Furthermore, due to problems such as camera lens distortion and shooting angle error in actual applications, the matching error between feature points is relatively large. Therefore, after calculating the homography matrix, the RANSAC algorithm can be used to remove outliers from the feature point matching pairs, so as to ensure that the estimated geometric transformation matrix is as accurate as possible. In addition, local features in the image can be combined, and local transformation can be used to further refine the image alignment, thereby improving the image alignment accuracy.

[0118] Through such settings, this solution can judge the complexity of image changes according to the contour similarity of feature points, and determine the image mapping relationship using different transformation models based on the complexity of image changes, so as to use the image mapping relationship for re-image alignment, which helps to further improve the accuracy of image alignment.

[0119] Figure 7 It is a schematic structural diagram of a pipeline leakage detection device provided by the present invention. This device can execute the pipeline leakage detection method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method. As Figure 7 shown, the device includes:

[0120] A target pipeline image determination module 310, configured to determine a target pipeline image according to the target pipeline image collected by the drone; wherein, the drone is equipped with a visible light camera and a thermal imaging camera, and the target pipeline image includes a target visible light image and a target thermal imaging image at the same time;

[0121] The first detection image determination module 320 is configured to perform pipeline leakage detection on the target visible light image by using a pre-trained first detection model to obtain a first detection image;

[0122] The second detection image determination module 330 is configured to perform pipeline leakage detection on the target thermal imaging image by using a pre-trained second detection model to obtain a second detection image;

[0123] The pipeline leakage detection result determination module 340 is configured to determine a pipeline leakage detection result according to the first detection image and the second detection image.

[0124] Optionally, the device further includes: a target pipeline image acquisition module, configured to:

[0125] Determine a target flight route of the unmanned aerial vehicle according to the target pipeline position information, and control the unmanned aerial vehicle to navigate according to the target flight route;

[0126] During the navigation of the unmanned aerial vehicle, use the visible light camera and the thermal imaging camera to capture the target pipeline at the same pan-tilt angle to obtain a plurality of candidate visible light images and a plurality of candidate thermal imaging images respectively;

[0127] Perform image stitching on the candidate visible light image and the candidate thermal imaging image at each same moment based on a preset stitching method to obtain a plurality of candidate stitched images;

[0128] Generate a target pipeline video according to the plurality of candidate stitched images in chronological order.

[0129] Optionally, the target pipeline image determination module 310 is configured to:

[0130] Determine a target frame extraction interval of the target pipeline video based on a preset frame extraction formula;

[0131] Perform frame extraction processing on the target pipeline video according to the target frame extraction interval to obtain a plurality of target stitched images;

[0132] Perform image segmentation on each of the target stitched images to obtain a plurality of target visible light images and target thermal imaging images at the same moment;

[0133] Wherein, the expression of the preset frame extraction formula is as follows:

[0134]

[0135] Wherein, N represents the frame extraction interval, T represents the pipeline leakage detection time, t represents the maximum detection time, and respectively represent the minimum frame extraction interval and the maximum frame extraction interval.

[0136] Optionally, the device further includes a detection image alignment module, and the detection image alignment module includes:

[0137] An initial image alignment unit, configured to perform initial image alignment on the first detection image and the second detection image according to the camera parameter information and shooting attitude information of the drone before determining the pipeline leakage detection result based on the first detection image and the second detection image, so as to obtain a first alignment result;

[0138] A re-image alignment unit, configured to perform re-image alignment on the first alignment result based on a feature matching algorithm to obtain a second alignment result;

[0139] A detection image update unit, configured to update the first detection image and the second detection image according to the second alignment result.

[0140] Optionally, the initial image alignment unit is configured to:

[0141] Determine the shooting angle of the visible light camera according to the camera parameter information of the visible light camera, and determine the shooting angle of the thermal imaging camera according to the camera parameter information of the thermal imaging camera;

[0142] Determine the shooting field of view of the visible light camera according to the shooting angle of the visible light camera and the shooting height of the drone, and determine the shooting field of view of the thermal imaging camera according to the shooting angle of the thermal imaging camera and the shooting height of the drone;

[0143] Determine a source image and a target image from the first detection image and the second detection image according to the shooting field of view of the visible light camera and the shooting field of view of the thermal imaging camera; wherein, the shooting field of view of the source image is smaller than the shooting field of view of the target image;

[0144] Determine the pixel field of view of the target image according to the shooting field of view of the target image and the camera resolution corresponding to the target image; wherein, the pixel field of view is used to characterize the field of view size occupied by each pixel point in the physical world in the image;

[0145] Determine a target size according to the shooting field of view of the source image and the pixel field of view of the target image, and crop the target image according to the target size to obtain a first image;

[0146] Adjust the image size of the first image according to the image size of the source image, and determine a first alignment result according to the source image and the second image.

[0147] Optionally, the re-image alignment unit is configured to:

[0148] Use a feature matching algorithm to determine the feature point matching pairs between the source image and the second image;

[0149] Determine the image mapping relationship between the source image and the second image according to the feature point matching pairs;

[0150] Correct the second image according to the image mapping relationship to obtain a third image, and determine a second alignment result according to the source image and the third image.

[0151] Optionally, the re-image alignment unit is further configured to:

[0152] After using the feature matching algorithm to determine the feature point matching pairs between the source image and the second image, determine the first feature point contour information according to the source image feature points in the feature point matching pairs, and determine the second feature point contour information according to the second image feature points in the feature point matching pairs;

[0153] Determine the contour similarity between the first feature point contour information and the second feature point contour information, and determine a target transformation model according to the contour similarity; wherein, the transformation model includes an affine transformation model and a perspective transformation model;

[0154] Determine the geometric transformation parameters between the source image and the second image according to the target transformation model and the feature point matching pairs; wherein, the geometric transformation parameters are an affine transformation matrix or a homography matrix;

[0155] Determine the geometric transformation parameters as the image mapping relationship between the source image and the second image.

[0156] Optionally, the re-image alignment unit is further configured to:

[0157] If the contour similarity is greater than a preset similarity, determine the target transformation model as an affine transformation model;

[0158] If the contour similarity is less than or equal to the preset similarity, determine the target transformation model as a perspective transformation model.

[0159] Optionally, the pipeline leakage detection result determination module 340 is configured to:

[0160] If there are leakage areas in both the first detection image and the second detection image, determine whether there is an overlap between the leakage areas in the first detection image and the second detection image;

[0161] If there is an overlap, determine that the target pipeline has a leakage and the warning level is the highest level.

[0162] Optionally, the pipeline leakage detection result determination module 340 is further configured to:

[0163] If there is a leakage area in the first detection image and no leakage area in the second detection image, determine, according to the image mapping relationship between the first detection image and the second detection image, the image area corresponding to the leakage area of the first detection image in the second detection image as the reference leakage area;

[0164] Take the image area within a preset range around the reference leakage area as the background area of the reference leakage area, and determine the first color feature information of the reference leakage area and the second color feature information of the background area;

[0165] If the difference between the first color feature information and the second color feature information is greater than a preset threshold, determine that the target pipeline has a leakage and the warning level is the intermediate level.

[0166] Optionally, the pipeline leakage detection result determination module 340 is further configured to:

[0167] If there is a leakage area in the second detection image and no leakage area in the first detection image, reduce the confidence level of the first detection model to a reference threshold to obtain an updated first detection model;

[0168] Use the updated first detection model to perform pipeline leakage detection on the target visible light image to obtain a reference detection image;

[0169] If there is a leakage area in the reference detection image and it overlaps with the leakage area in the second detection image, determine that the target pipeline has a leakage and the warning level is the lowest level.

[0170] Optionally, the device further includes: a pipeline leakage location positioning module, configured to:

[0171] After determining the pipeline leakage detection result according to the first detection image and the second detection image, determine the target pipeline image with leakage in the target pipeline as the pipeline leakage image according to the pipeline leakage detection result, and obtain the target drone longitude and latitude information when the pipeline leakage image is taken;

[0172] Determine one of the multiple preset fire hydrants deployed around the target pipeline as the target fire hydrant, and obtain the target identification information of the target fire hydrant; wherein, the longitude and latitude information of the target fire hydrant has the smallest difference from the target drone longitude and latitude information;

[0173] Determine the relative angle and relative distance between the longitude and latitude information of the target UAV and the longitude and latitude information of the target fire hydrant, and locate the pipeline leakage position according to the target identification information and the relative angle and relative distance.

[0174] A pipeline leakage detection device provided by an embodiment of the present invention can execute a pipeline leakage detection method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0175] Figure 8 FIG. shows a schematic structural diagram of an electronic device 10 that can be used to implement embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processors, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0176] As Figure 8 shown, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. The memory stores a computer program executable by the at least one processor. The processor 11 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0177] Multiple components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disc, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0178] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the pipeline leakage detection method.

[0179] In some embodiments, the pipeline leakage detection method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the pipeline leakage detection method described above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute the pipeline leakage detection method by any other suitable means (e.g., by means of firmware).

[0180] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuitry, integrated circuit systems, field-programmable gate arrays (FPGA), application-specific integrated circuits (ASIC), application-specific standard products (ASSP), system-on-a-chip (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0181] The computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processors of a general-purpose computer, a special-purpose computer, or other programmable data processing devices, such that when the computer programs are executed by the processors, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0182] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0183] In order to provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, speech input, or tactile input).

[0184] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0185] A computing system may include a client and a server. The client and the server are generally far from each other and usually interact via a communication network. The relationship between the client and the server is created by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, solving the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0186] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is made herein.

[0187] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A pipeline leakage detection method, characterized in that, The method includes: Determining a target pipeline image based on the target pipeline images collected by the unmanned aerial vehicle (UAV); wherein, the UAV is equipped with a visible light camera and a thermal imaging camera, and the target pipeline image includes a target visible light image and a target thermal imaging image at the same moment; Performing pipeline leakage detection on the target visible light image by using a pre-trained first detection model to obtain a first detection image; Performing pipeline leakage detection on the target thermal imaging image by using a pre-trained second detection model to obtain a second detection image; Determining a pipeline leakage detection result according to the first detection image and the second detection image.

2. The method according to claim 1, wherein The process of collecting the target pipeline images includes: Determining a target flight route of the UAV according to the target pipeline position information, and controlling the UAV to navigate according to the target flight route; During the navigation of the UAV, using the visible light camera and the thermal imaging camera to capture the target pipeline at the same pan-tilt angle, respectively obtaining a plurality of candidate visible light images and a plurality of candidate thermal imaging images; Performing image stitching on each candidate visible light image and candidate thermal imaging image at the same moment based on a preset stitching method to obtain a plurality of candidate stitched images; Generating a target pipeline video according to the plurality of candidate stitched images in chronological order.

3. The method according to claim 2, wherein Determining a target pipeline image based on the target pipeline images collected by the UAV includes: Determining a target frame extraction interval of the target pipeline video based on a preset frame extraction formula; Performing frame extraction processing on the target pipeline video according to the target frame extraction interval to obtain a plurality of target stitched images; Performing image segmentation on each of the target stitched images to obtain a plurality of target visible light images and target thermal imaging images at the same moment; Wherein, the expression of the preset frame extraction formula is as follows: Among them, N represents the frame extraction interval, T represents the pipeline leakage detection time, and t represents the maximum detection time. and represent the minimum frame extraction interval and the maximum frame extraction interval respectively.

4. The method according to claim 1, wherein Before determining the pipeline leakage detection result according to the first detection image and the second detection image, it further includes: Performing initial image alignment on the first detection image and the second detection image according to the camera parameter information and shooting attitude information of the UAV to obtain a first alignment result; Performing secondary image alignment on the first alignment result based on a feature matching algorithm to obtain a second alignment result; Updating the first detection image and the second detection image according to the second alignment result.

5. The method according to claim 4, characterized in that, Performing initial image alignment on the first detection image and the second detection image according to the camera parameter information and shooting attitude information of the UAV to obtain a first alignment result, including: Determining the shooting angle of the visible light camera according to the camera parameter information of the visible light camera, and determining the shooting angle of the thermal imaging camera according to the camera parameter information of the thermal imaging camera; Determining the shooting field of view of the visible light camera according to the shooting angle of the visible light camera and the UAV shooting height, and determining the shooting field of view of the thermal imaging camera according to the shooting angle of the thermal imaging camera and the UAV shooting height; Determining a source image and a target image from the first detection image and the second detection image according to the shooting field of view of the visible light camera and the shooting field of view of the thermal imaging camera; wherein, the shooting field of view of the source image is smaller than the shooting field of view of the target image; Determine the pixel field of view of the target image according to the shooting field of view of the target image and the camera resolution corresponding to the target image; wherein, the pixel field of view is used to characterize the field of view size occupied by each pixel point in the physical world; Determine the target size according to the shooting field of view of the source image and the pixel field of view of the target image, and crop the target image according to the target size to obtain a first image; Adjust the image size of the first image according to the image size of the source image to obtain a second image, and determine a first alignment result according to the source image and the second image.

6. The method according to claim 5, wherein Perform secondary image alignment on the first alignment result based on a feature matching algorithm to obtain a second alignment result, including: Use a feature matching algorithm to determine the feature point matching pairs between the source image and the second image; Determine the image mapping relationship between the source image and the second image according to the feature point matching pairs; Correct the second image according to the image mapping relationship to obtain a third image, and determine a second alignment result according to the source image and the third image.

7. The method according to claim 6, wherein After using a feature matching algorithm to determine the feature point matching pairs between the source image and the second image, it further includes: Determine the first feature point contour information according to the source image feature points in the feature point matching pairs, and determine the second feature point contour information according to the second image feature points in the feature point matching pairs; Determine the contour similarity between the first feature point contour information and the second feature point contour information, and determine the target transformation model according to the contour similarity; wherein, the transformation model includes an affine transformation model and a perspective transformation model; Correspondingly, determining the image mapping relationship between the source image and the second image according to the feature point matching pairs includes: Determine the geometric transformation parameters between the source image and the second image according to the target transformation model and the feature point matching pairs; wherein, the geometric transformation parameters are an affine transformation matrix or a homography matrix; Determine the geometric transformation parameters as the image mapping relationship between the source image and the second image.

8. The method according to claim 7, characterized in that, Determining the target transformation model according to the contour similarity includes: If the contour similarity is greater than a preset similarity, determine the target transformation model as an affine transformation model; If the contour similarity is less than or equal to the preset similarity, determine the target transformation model as a perspective transformation model.

9. The method according to any one of claims 1-8, characterized in that, Determine the pipeline leakage detection result according to the first detection image and the second detection image, including: If there are leakage areas in both the first detection image and the second detection image, determine whether there is an overlap between the leakage areas in the first detection image and the second detection image; If there is an overlap, determine that the target pipeline has a leak.

10. The method according to any one of claims 1-8, characterized in that, Determine the pipeline leakage detection result according to the first detection image and the second detection image, including: If there is a leakage area in the first detection image and no leakage area in the second detection image, determine, according to the image mapping relationship between the first detection image and the second detection image, the image area in the second detection image corresponding to the leakage area of the first detection image as the reference leakage area; Take the image area within a preset range around the reference leakage area as the background area of the reference leakage area, and determine the first color feature information of the reference leakage area and the second color feature information of the background area; If the difference between the first color feature information and the second color feature information is greater than a preset threshold, determine that there is a leakage in the target pipeline.

11. The method according to any one of claims 1-8, characterized in that, Determine the pipeline leakage detection result according to the first detection image and the second detection image, including: If there is a leakage area in the second detection image and no leakage area in the first detection image, reduce the confidence level of the first detection model to a reference threshold to obtain an updated first detection model; Use the updated first detection model to perform pipeline leakage detection on the target visible light image to obtain a reference detection image; If there is a leakage area in the reference detection image and it overlaps with the leakage area in the second detection image, determine that there is a leakage in the target pipeline.

12. The method according to claim 1, wherein After determining the pipeline leakage detection result according to the first detection image and the second detection image, it further includes: Determine the target pipeline image with a leakage in the target pipeline as the pipeline leakage image according to the pipeline leakage detection result, and obtain the longitude and latitude information of the target drone when the pipeline leakage image is taken; Determine one of the multiple preset fire hydrants deployed around the target pipeline as the target fire hydrant, and obtain the target identification information of the target fire hydrant; wherein, the longitude and latitude information of the target fire hydrant has the smallest difference from the longitude and latitude information of the target drone; Determine the relative angle and relative distance between the longitude and latitude information of the target drone and the longitude and latitude information of the target fire hydrant, and locate the pipeline leakage position according to the target identification information and the relative angle and relative distance.

13. A pipeline leakage detection device, characterized in that, The device includes: A target pipeline image determination module, configured to determine a target pipeline image according to the target pipeline image collected by the drone; wherein, the drone is equipped with a visible light camera and a thermal imaging camera, and the target pipeline image includes a target visible light image and a target thermal imaging image at the same time; A first detection image determination module, configured to use a pre-trained first detection model to perform pipeline leakage detection on the target visible light image to obtain a first detection image; A second detection image determination module, configured to use a pre-trained second detection model to perform pipeline leakage detection on the target thermal imaging image to obtain a second detection image; A pipeline leakage detection result determination module, configured to determine the pipeline leakage detection result according to the first detection image and the second detection image.

14. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, enables the at least one processor to execute the pipeline leakage detection method according to any one of claims 1-12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for implementing the pipeline leakage detection method according to any one of claims 1-12 when the computer instructions are executed by a processor.

Images