Method and system for routing inspection of cable channel by unmanned aerial vehicle

Abstract

The invention discloses a method and system for routing inspection of a cable channel by an unmanned aerial vehicle, and the method comprises the steps: carrying out the routing inspection of the cable channel in construction through the unmanned aerial vehicle at a channel construction stage, and carrying out the geographic position calculation and modeling of the cable channel based on a language prompt word coding module and closed operation; planning a cable channel inspection route based on the established cable channel model; in the inspection stage, the unmanned aerial vehicle autonomously flies according to a planned route and shoots real-time images, and hidden danger target recognition and effective alarm analysis of the position and the moving direction are carried out on each frame of image. The route planning efficiency is high, and the polling effect and timeliness are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of cable channel inspection, and relates to a method and system for inspecting a cable channel by using a drone. Background Art

[0002] Cable channels are one of the important objects of power system inspection. Timely discovery of channel hidden dangers is crucial to ensuring power safety. With the maturity of drone technology and products, they are increasingly used in cable channel inspection scenarios to replace manual field inspections, which can improve inspection efficiency and the timeliness of discovering hidden dangers.

[0003] However, there are still some problems with drone inspection of cable channels, which are mainly reflected in two aspects. First, the route cannot be automatically generated, but it needs the help of professionals who are familiar with the location of the cable channel and drone pilots to manually define the inspection route; second, the drone is only used as a means of data collection during flight, and the hidden danger analysis is carried out after the drone flight. The former leads to a large workload for route planning, prone to errors, and low efficiency; the latter leads to the separation of flight and analysis, poor inspection results, and certain delays. Summary of the invention

[0004] In order to address the deficiencies in the prior art, the present invention provides a method and system for inspecting cable channels using drones, which automatically calculates and models the geographic location of the cable channel during the channel construction phase, thereby automatically planning the route, and uses computer vision algorithms to identify potential hidden danger targets in real time during drone inspections, and provides effective warnings through position and movement analysis.

[0005] The present invention adopts the following technical solution.

[0006] A first aspect of the present invention provides a method for inspecting a cable channel using a drone, comprising:

[0007] During the channel construction phase, drones are used to inspect the cable channel under construction, and the geographical location of the cable channel is calculated and modeled based on the language prompt word encoding module and closed operations;

[0008] Based on the established cable channel model, plan the cable channel inspection route;

[0009] During the inspection phase, the drone flies autonomously according to the planned route and takes real-time images, identifying potential hazards and conducting alarm analysis on the location and movement direction of each frame.

[0010] Preferably, during the channel construction phase, a drone is used to inspect the cable channel under construction, and the geographical location of the cable channel is calculated and modeled based on a language prompt word encoding module and a closed operation, specifically including:

[0011] The drone inspects the cable channel under construction, and its gimbal camera takes real-time photos of the excavated cable channel to obtain flight videos;

[0012] Extract each frame of the flight video and perform image segmentation based on the language prompt word encoding module to obtain the image mask after the corresponding frame segmentation;

[0013] Based on the closing operation, the connectivity analysis of the mask of each frame is performed and the maximum connected domain is fitted into a quadrilateral to obtain the quadrilateral of the channel captured by each frame of the image;

[0014] Based on the GPS position of the drone and the camera parameters, the longitude and latitude of the quadrilateral of the channel captured by each frame of the image are calculated;

[0015] The channel positions of adjacent frame images are smoothly connected according to the longitude and latitude to obtain the edge of the channel.

[0016] Preferably, the step of extracting each frame of the flight video and performing image segmentation based on the language prompt word encoding module to obtain an image mask after the corresponding frame segmentation includes:

[0017] The SAM model inputs an image and a prompt word "excavated tunnel" and outputs a segmented image mask, in which the pixel value of the excavated tunnel category is 1, otherwise it is 0; the SAM model uses a pre-trained language prompt word encoding module to recognize the prompt word "excavated tunnel", specifically:

[0018] The language prompt word encoding module tokenizes the text and converts it into a word vector through an embedding matrix, and uses a cosine function to generate a position encoding vector; the position encoding vector and the word vector are added and input into a self-attention encoding layer, which learns the phrase relationship of the input vector through multi-head self-attention.

[0019] Preferably, the method of performing connectivity analysis on the mask of each frame based on a closing operation and fitting the maximum connected domain into a quadrilateral to obtain the quadrilateral of the channel captured by each frame of the image comprises:

[0020] Perform image morphological closing operation on the mask of the current frame to segment a more complete tunnel; perform connectivity analysis on the mask of the current frame to find the largest connected domain, and fit the connected domain into a quadrilateral. If the parameters of the fitted quadrilateral are within the set range, it is retained, otherwise the current frame is skipped and processing of the next frame continues.

[0021] Preferably, the kernel initialization method used in the closing operation is:

[0022] Randomly select more than 10% of the video frames to form a missed detection calibration data set, use the SAM model to segment, and manually review and mark the missed detection areas based on the segmentation results. When marking, mark the missed detection areas as 1 and other areas as 0;

[0023] Perform missed detection statistics on the marking results of the missed detection calibration data set, count the diameter size distribution of all missed detection areas, and obtain the corresponding histogram;

[0024] After Gaussian smoothing of the histogram, the diameter of the peak position is used as the kernel size, and the kernel is initialized with this size to perform a closing operation on the mask.

[0025] Preferably, planning the cable channel inspection route based on the established cable channel model specifically includes:

[0026] One side of the channel in the cable channel model is selected as route 1, and the other side is selected as route 2. Waypoints are taken on route 1 at set distance intervals. Assuming that any waypoint is M, the point closest to point M on route 2 is calculated and set as N. Then the center point of M and N is calculated, and curve fitting is performed on all center points to obtain a smooth inspection route; and the minimum inspection height of the drone is set for the obtained inspection route.

[0027] Preferably, during the inspection phase, the drone autonomously flies according to the planned route and captures real-time images, and performs hidden danger target identification and position and movement direction alarm analysis on each frame of the image, including:

[0028] For each frame of the image, a target detection algorithm is used to identify potential targets in the image, including various types of construction machinery and vehicles; when the potential targets are identified, an alarm analysis of the position and movement direction is started.

[0029] Preferably, the alarm analysis method of the location is as follows:

[0030] For each frame of image, obtain the GPS coordinate information of the drone when the frame was taken, and calculate the GPS coordinate information of each pixel in the frame according to the gimbal attitude angle and the focal length of the camera when shooting;

[0031] If the GPS of a pixel is within the set channel alarm range, it is marked as 1, otherwise it is marked as 0, and the alarm binary image of the frame image is obtained;

[0032] For each identified hidden danger target, a rectangular box is used to represent its position in the image. The pixel value in the rectangular box is marked as 1, otherwise it is marked as 0, and the target binary map of the image is obtained;

[0033] Perform a union operation on the alarm binary image and the target binary image, and output an intersection mask image;

[0034] If there is a point with a value of 1 in the intersection mask map, there is a hidden danger target within the warning range, and an alarm is issued.

[0035] Preferably, after the position analysis, a moving direction alarm analysis is performed for hidden danger targets that are not within the alarm range, and the specific analysis steps are:

[0036] For any hidden danger target, track the position change of the hidden danger target in continuous image frames to obtain the pixel distance moved by the hidden danger target in the image. According to the time interval between images and the pixel distance moved, the moving speed of the hidden danger relative to the UAV can be calculated.

[0037] The flight control system obtains the moving speed of the drone body, and the moving speed of the hidden danger relative to the ground is calculated based on the moving speed of the drone body and the moving speed of the hidden danger relative to the drone; when the moving speed of the hidden danger relative to the ground is 0, the hidden danger target is judged to be in a stationary state, otherwise it is in a moving state;

[0038] For hidden danger targets in a moving state, continue to calculate their components in the direction orthogonal to the channel along the UAV. If the component is negative, it is determined that the moving direction is away from the channel. If the component is positive, it is determined that the moving direction is close to the channel, and an alarm is issued. If the component is zero, it is determined that the hidden danger is stationary at this time and is not in the alarm area.

[0039] A second aspect of the present invention is a cable channel inspection system using a drone, comprising:

[0040] The channel modeling module is used to use drones to inspect the cable channel under construction during the channel construction phase, and calculate and model the geographical location of the cable channel based on the language prompt word encoding module and closed operations;

[0041] The route planning module is used to plan the cable channel inspection route based on the established cable channel model;

[0042] The recognition and analysis module is used for the UAV to fly autonomously according to the planned route and take real-time images, identify potential danger targets for each frame of the image, and conduct alarm analysis on the position and movement direction.

[0043] A terminal includes a processor and a storage medium, wherein the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the method.

[0044] A computer-readable storage medium stores a computer program, which implements the steps of the method when executed by a processor.

[0045] The beneficial effects of the present invention are as follows:

[0046] The present invention automatically calculates and models the geographical location of the cable channel during the channel construction stage. By modeling the geographical location of the cable channel in advance, the route can be automatically generated. The route planning efficiency is high, and the problem that the channel is buried underground after completion and the position is invisible, which causes the subsequent drone inspection to be unable to accurately determine whether various targets that need to be identified are within the protected range of the channel, can be effectively solved;

[0047] The present invention can identify potential hidden dangers in real time during drone inspections, and perform position and movement analysis on mobile targets to identify whether the targets are approaching cable channels, thereby providing advance warnings of impending risks, avoiding further harm caused by hidden dangers, and improving the effectiveness and timeliness of inspection hidden danger identification.

[0048] The present invention improves the prompt word encoding module of the SAM model and proposes a language prompt word encoding module, which can segment the "dug tunnel" area in the image through the language prompt words.

[0049] The present invention proposes an adaptive closing operation kernel initialization method, which solves the problem of poor closing operation effect caused by large morphological differences of missed objects, thereby improving the integrity of tunnel extraction in the next step of the maximum connected domain. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of the method of the present invention;

[0051] Figure 2 This is a schematic diagram for calculating the location of the cable channel;

[0052] Figure 3 Generate schematic diagrams for drone inspection routes. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The embodiments described in this application are only part of the embodiments of the present invention, not all of them. Based on the spirit of the present invention, other embodiments obtained by ordinary technicians in this field without creative work are all within the scope of protection of the present invention.

[0054] Embodiment 1 of the present invention provides a method for inspecting a cable channel by using a drone, which uses an algorithm to generate the location of the channel during the construction phase and automatically plans the drone inspection route. During the inspection phase, an algorithm is used to automatically analyze potential hazards and generate effective alarms. Figure 1 As shown, the method specifically includes:

[0055] S1: During the channel construction phase, drones are used to inspect the cable channel under construction and calculate and model the geographical location of the cable channel;

[0056] Further preferably, during the channel construction phase, a drone is used to inspect the cable channel under construction, and the geographical location of the cable channel is automatically modeled to obtain the location of the channel. The cable channel modeling method is as follows:

[0057] S1.1: The drone inspects the cable channel under construction and takes photos of the excavated cable channel to obtain a flight video;

[0058] During the construction phase of the cable channel, the channel is dug to form a cable tunnel. The drone pilot controls the drone to fly along the cable channel under construction. During the flight, the drone flies vertically along the top of the tunnel with the nose facing the tunnel, starts the drone GPS positioning, and uses the gimbal camera mounted on the drone to shoot the tunnel, that is, the cable channel dug on the ground, and stores the shot images as a video to obtain a flight video.

[0059] After the flight, use method 1 to analyze the video, segment the tunnel in each frame, and combine GPS position and gimbal attitude, camera parameters, channel width and other information to generate edge curves on both sides of the tunnel with real GPS information. The specific steps are as follows:

[0060] S1.2: Extract each frame of the flight video and perform image segmentation to obtain the segmented image mask of the corresponding frame; specifically:

[0061] Extract the images in the flight video. For each frame of the extracted image, use SAM and other segmentation models, input the image and the prompt word "dug tunnel", and the model outputs the image mask. The pixel value belonging to the dug tunnel category is 1, otherwise it is 0.

[0062] Because the original SAM model can only accept three forms of prompt word input, such as points, rectangular boxes, and binary segmentation areas, and does not support category prompt words described in natural language, the present invention improves the prompt word encoding module of the SAM model as follows:

[0063] First, the text is tokenized and converted into a vector through an embedding matrix, so that each word is mapped to a vector space of fixed dimension, where the vector space dimension is specified to be 512.

[0064] Since the Transformer does not use recursive or convolutional structures, a cosine function is used to generate positional encodings to capture the position information of words.

[0065] Add the positional encoding vector and the word vector and input them into the self-attention encoding layer, which consists of multi-head self-attention with 4 heads, allowing the model to learn the relationships among the 4 phrases "by", "dug out", "of", and "tunnel" in the prompt word. Designing the number of heads of the encoder based on such prior information can effectively accelerate the convergence speed during model fine-tuning.

[0066] After replacing the original prompt word encoding module of SAM with this prompt word encoding module to form a new model, the new model needs to be fine-tuned. Specifically:

[0067] Manually annotate no less than one thousand segmented images of tunnel areas using an annotation tool as the fine-tuning dataset. Freeze the parameters of the image encoder and Mask decoder of the SAM model, input the fine-tuning dataset images and annotations, and train the newly replaced prompt word encoder. Stop training when the model converges to obtain a new model that can segment the "dug out tunnel" area in the image through a language prompt word.

[0068] S1.3: Conduct connectivity analysis on the mask of each frame and fit the largest connected component to a quadrilateral to obtain the quadrilateral of the channel captured in each frame of the picture; specifically:

[0069] Perform a closing operation of image morphology on the mask to make the tunnel pixels in the mask more coherent and improve the smoothness of the segmentation edge, so as to segment a more complete tunnel. When performing the closing operation of image morphology, a major problem is how to give an appropriate size of the closing operation kernel. To determine the appropriate kernel size, it is necessary to uniformly sample image samples from the dataset according to time, and conduct a missed detection statistics on the masks segmented by SAM. Statistically analyze the diameter size distribution of all missed detection areas to obtain a histogram. Smooth the histogram with a Gaussian function and use the diameter at the peak position as the kernel size. Initialize the kernel with this size and then perform a closing operation on the mask. Specifically as follows:

[0070] Randomly select more than 10% of the video frames to form a missed detection calibration dataset, segment it using the SAM model, and manually review and mark the missed detection areas according to the segmentation results. When marking, mark the missed detection areas as 1 and other areas as 0;

[0071] Conduct missed detection statistics on the marking results of the missed detection calibration dataset, statistically analyze the diameter size distribution of all missed detection areas to obtain the corresponding histogram;

[0072] After Gaussian smoothing of the histogram, the diameter of the peak position is used as the kernel size, and the kernel is initialized with this size to perform a closing operation on the mask; this initialization method is a domain-adaptive closing operation kernel initialization method, which solves the problem of poor closing operation effect caused by large morphological differences of missed objects, thereby improving the integrity of the tunnel extraction in the next step of the maximum connected domain.

[0073] Continue to perform connectivity analysis on the processed mask to find the largest connected domain and fit the connected domain into a quadrilateral. Set a reasonable range for the area, long side, and short side. If the fitted quadrilateral parameters are within a reasonable range, keep it. Otherwise, skip the current frame and continue processing the next frame.

[0074] S1.4: Calculate the longitude and latitude of the quadrilateral of the channel captured in each frame based on the GPS position of the drone and the camera parameters;

[0075] 1) Calculate the pixel value ratio of the channel design width and the fitted short side to obtain the scale ratio s of the frame image and the actual physical world, that is, pixels / meter.

[0076] 2) According to the camera focal length value f, the scale ratio s and the triangle similarity, the straight-line distance d = f / s between the camera and the photographed channel can be calculated.

[0077] 3) According to the known angle α between the gimbal and the ground when the gimbal takes the image, the height h=d*cos(α) of the gimbal and the ground is calculated by trigonometric function.

[0078] 4) Continue to process the fitted quadrilateral. According to the GPS position of the drone, the angle between the gimbal and the ground is α, and the straight-line distance d, the trigonometric function is used to calculate that the position corresponding to the center of the image and the straight-line distance of the drone are approximately d*sin(α). Combining the drone's position and navigation angle, the GPS of the center of the image can be approximately calculated. Then, according to the scale ratio s and the distance and angle between the vertex and the center of the quadrilateral, the latitude and longitude of the quadrilateral, that is, the longitude and latitude of the cable channel, can be calculated. Figure 2 For example, when the drone takes the first frame, its position is P, and the calculated positions of the channels are A, B, C, and D respectively.

[0079] S1.5: Smoothly connect the channel positions of adjacent frame images according to the longitude and latitude to obtain the edge of the channel.

[0080] After calculating the longitude and latitude of the quadrilateral of the channel captured by each frame, the channel positions of adjacent frame images are smoothly connected. Figure 2As shown, the drone is located at position P when taking the first frame, and is located at position Q when taking the next frame. When at position Q, the GPS positions of the four vertices of the channel are obtained using the above method, which are E, F, G, and H, respectively.

[0081] A, D, E, H are one side of the channel, and B, C, F, G are the other side of the channel.

[0082] The position points on each side are interpolated, such as cubic spline interpolation, to obtain a smooth curve on one side, indicating the edge of the channel.

[0083] S2: Plan the cable channel inspection route based on the cable channel model;

[0084] More preferably, if Figure 3 As shown in the figure, after calculating the edge curves on both sides of the tunnel, the route generation method is used to automatically plan an available drone inspection route that meets the height requirements. If the tunnel is discontinuous, that is, there are cable channels spanning multiple locations, multiple inspection routes are generated, as follows:

[0085] S2.1: According to S1, the GPS location information of the cable channel is obtained. In this step, a drone inspection route for inspecting along the channel is automatically generated based on the location information.

[0086] Select one side of the channel generated in S1 as route 1 and the other side as route 2. Take waypoints on route 1 at a certain distance interval. Assume that any one of the waypoints is M. Use Newton's method or other numerical optimization algorithms to calculate the point on route 2 closest to point M, set it as N, and then calculate the center position O of M and N. Perform curve fitting on all center points to obtain a smooth inspection route.

[0087] S2.2: Set the minimum inspection height h of the drone for the inspection route obtained in S2.1.

[0088] In addition to GPS location, the inspection route of the drone should also include altitude information. The inspection range of the channel should be set to be within d meters on both sides of the channel. According to the focal length of the drone camera and the sensor imaging size during the inspection, the minimum altitude h that should be maintained in order to capture the complete inspection range of the channel can be calculated using triangle similarity.

[0089] S3: During the inspection phase, the drone autonomously flies according to the route planned by S2 and takes real-time images, identifying potential hazards and conducting effective alarm analysis for each frame of the image.

[0090] Preferably, once the inspection route is generated, the drone can autonomously fly along the route, i.e., conduct inspection. When the drone inspects along the route, it uses a real-time analysis algorithm to analyze the video footage captured by the camera, identify potential targets, and issue effective warnings based on information such as the target location and moving direction. If there is a target that meets the requirements, an alarm message is generated and sent to the monitoring system. The alarm message contains the drone location information, target information, and image screen, as follows:

[0091] S3.1: During the inspection phase, the drone flies autonomously according to the route planned in S2, and the gimbal camera captures real-time images. For each frame, the target detection algorithm is used to identify potential targets in the image, including excavators, bulldozers, pile drivers, cranes and other construction machinery and vehicles that may damage the cable channel.

[0092] S3.2: Once the target is identified, start effective warning analysis.

[0093] a) Position analysis. For each frame of image, obtain the coordinate information (lat, lon, alt) when the frame was taken, and calculate the GPS coordinate information (latpixel, lonpixel, alt) of each pixel point (xp, yp) in the frame of image based on the gimbal attitude angle and the focal length of the camera when shooting (the height does not change much);

[0094] Pixel point GPS information calculation process:

[0095] GPS location of the drone: (lat, lon, alt);

[0096] Gimbal attitude angles: yaw angle (Yaw, ψ), pitch angle (Pitch, θ), roll angle (Roll, φ);

[0097] Camera parameters: focal length f, sensor size (width w and height h), image resolution (pixel width W and height H);

[0098] Earth's radius: R earth

[0099] The calculation steps are as follows:

[0100] 1) Convert pixel coordinates to image coordinates

[0101] ·Let the coordinates of a pixel be (xp, yp)

[0102] Transform to the center of the image coordinate system:

[0103]

[0104]

[0105] 2) Convert camera coordinates to world coordinates:

[0106] Project the image coordinates to the camera coordinate system:

[0107] X=x c

[0108] Y=y c

[0109] Z=f

[0110] Consider the gimbal posture:

[0111]

[0112] The rotation matrix R is formed by the yaw, pitch and roll angles:

[0113] R=R yaw (ψ)·R pitch (θ)·R roll (φ)

[0114] 3) Convert world coordinates to geographic coordinates

[0115] Convert camera coordinates to GPS coordinates, assuming the drone is on the ground:

[0116]

[0117] Final GPS coordinates:

[0118] lat pixel = lat + Δ lat

[0119] lon pixel =lon+Δlon

[0120] If the GPS of a pixel point is within the effective alarm range of the channel, it is marked as 1, otherwise it is marked as 0, and the effective alarm binary map of the frame image is obtained. For each identified target, its position in the image is represented by a rectangular box, and the pixel value in the rectangular box is marked as 1, otherwise it is marked as 0, and the target binary map of the frame image is obtained. Perform a AND operation on the effective alarm binary map and the target binary map, and output the intersection mask map. If there is a point with a value of 1 in the intersection mask map, there is a target within the effective alarm range, and an alarm should be issued.

[0121] b) Motion analysis. For targets that are not within the effective warning range, a motion analysis is performed. If the target moves toward the channel, it is determined to be a risky target. The specific analysis steps are as follows: For any target, use the optical flow algorithm to track the position change of the hidden danger target in continuous image frames, and the pixel distance of the hidden danger target in the image can be obtained. The moving speed of the hidden danger relative to the drone can be calculated based on the time interval between images and the moving pixel distance. In the rectangular box in each frame of the image, the optical flow algorithm is used to calculate the moving speed within the rectangular box.

[0122] The flight control system obtains the moving speed of the drone body (relative to the ground), and calculates the moving speed of the hidden danger relative to the ground based on the moving speed of the drone body and the moving speed of the hidden danger relative to the drone; when the moving speed of the hidden danger relative to the ground is 0, the hidden danger target is judged to be in a stationary state, otherwise it is in a moving state;

[0123] For targets in a moving state, continue to calculate their components in the direction orthogonal to the channel of the UAV, that is, perform a dot product operation on the hidden danger's moving speed and the unit vector in the direction orthogonal to the channel. If it is negative, it is judged to be far away from the channel. If it is positive, it is judged to be close to the channel and an alarm should be issued. If the component is zero, it is judged that the hidden danger is stationary at this time and is not in the effective alarm area.

[0124] Embodiment 2 of the present invention provides a cable channel inspection system using a drone, the system comprising:

[0125] The channel modeling module is used to use drones to inspect the cable channel under construction during the channel construction phase, and calculate and model the geographical location of the cable channel based on the language prompt word encoding module and closed operations;

[0126] The route planning module is used to plan the cable channel inspection route based on the established cable channel model;

[0127] The recognition and analysis module is used for the UAV to fly autonomously according to the planned route and take real-time images, identify potential hazards in each frame of the image, and conduct effective warning analysis of the location and movement direction.

[0128] Embodiment 3 of the present invention provides a terminal, including a processor and a storage medium; the storage medium is used to store instructions;

[0129] The processor is configured to operate according to the instructions to execute the steps of the method.

[0130] Embodiment 4 of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method when executed by a processor.

[0131] The beneficial effects of the present invention are as follows:

[0132] The present invention automatically calculates and models the geographical location of the cable channel during the channel construction stage. By modeling the geographical location of the cable channel in advance, the route can be automatically generated. The route planning efficiency is high, and the problem that the channel is buried underground after completion and the position is invisible, which causes the subsequent drone inspection to be unable to accurately determine whether various targets that need to be identified are within the protected range of the channel, can be effectively solved;

[0133] The present invention can identify potential hidden dangers in real time during drone inspections, and perform position and movement analysis on mobile targets to identify whether the targets are approaching cable channels, thereby providing advance warnings of impending risks, avoiding further harm caused by hidden dangers, and improving the effectiveness and timeliness of inspection hidden danger identification.

[0134] The present invention improves the prompt word encoding module of the SAM model and proposes a language prompt word encoding module, which can segment the "dug tunnel" area in the image through the language prompt words.

[0135] The present invention proposes an adaptive closing operation kernel initialization method, which solves the problem of poor closing operation effect caused by large morphological differences of missed objects, thereby improving the integrity of tunnel extraction in the next step of the maximum connected domain.

[0136] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0137] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0138] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0139] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for inspecting a cable channel using a drone, characterized in that: The method comprises: During the channel construction phase, drones are used to inspect the cable channel under construction, and the geographical location of the cable channel is calculated and modeled based on the language prompt word encoding module and closed operations; Based on the established cable channel model, plan the cable channel inspection route; During the inspection phase, the drone flies autonomously according to the planned route and takes real-time images, identifying potential hazards and conducting alarm analysis on the location and movement direction of each frame.

2. The method for inspecting a cable channel by using a drone according to claim 1, characterized in that: During the channel construction phase, a drone is used to inspect the cable channel under construction, and the geographical location of the cable channel is calculated and modeled based on a language prompt word encoding module and a closed operation, specifically including: The drone inspects the cable channel under construction, and its gimbal camera takes real-time photos of the excavated cable channel to obtain flight videos; Extract each frame of the flight video and perform image segmentation based on the language prompt word encoding module to obtain the image mask after the corresponding frame segmentation; Based on the closing operation, the connectivity analysis of the mask of each frame is performed and the maximum connected domain is fitted into a quadrilateral to obtain the quadrilateral of the channel captured by each frame of the image; Based on the GPS position of the drone and the camera parameters, the longitude and latitude of the quadrilateral of the channel captured by each frame of the image are calculated; The channel positions of adjacent frame images are smoothly connected according to the longitude and latitude to obtain the edge of the channel.

3. The method for inspecting a cable channel by using a drone according to claim 2, characterized in that: The method of extracting each frame of the flight video and performing image segmentation based on the language prompt word encoding module to obtain an image mask after the corresponding frame segmentation includes: The SAM model inputs an image and a prompt word "excavated tunnel" and outputs a segmented image mask, in which the pixel value of the excavated tunnel category is 1, otherwise it is 0; the SAM model uses a pre-trained language prompt word encoding module to recognize the prompt word "excavated tunnel", specifically: The language prompt word encoding module tokenizes the text and converts it into a word vector through an embedding matrix, and uses a cosine function to generate a position encoding vector; the position encoding vector and the word vector are added and input into a self-attention encoding layer, which learns the phrase relationship of the input vector through multi-head self-attention.

4. The method for inspecting a cable channel by using a drone according to claim 2, characterized in that: The method performs connectivity analysis on the mask of each frame based on the closing operation and fits the maximum connected domain into a quadrilateral to obtain the quadrilateral of the channel captured by each frame of the image, including: Perform image morphological closing operation on the mask of the current frame to segment a more complete tunnel; perform connectivity analysis on the mask of the current frame to find the largest connected domain, and fit the connected domain into a quadrilateral. If the parameters of the fitted quadrilateral are within the set range, it is retained, otherwise the current frame is skipped and processing of the next frame continues.

5. The method for inspecting a cable channel by using a drone according to claim 4, characterized in that: The kernel initialization method used in the closing operation is: Randomly select more than 10% of the video frames to form a missed detection calibration data set, use the SAM model to segment, and manually review and mark the missed detection areas based on the segmentation results. When marking, mark the missed detection areas as 1 and other areas as 0; Perform missed detection statistics on the marking results of the missed detection calibration data set, count the diameter size distribution of all missed detection areas, and obtain the corresponding histogram; After Gaussian smoothing of the histogram, the diameter of the peak position is used as the kernel size, and the kernel is initialized with this size to perform a closing operation on the mask.

6. The method for inspecting a cable channel by using a drone according to claim 1, characterized in that: The cable channel inspection route is planned based on the established cable channel model, specifically including: One side of the channel in the cable channel model is selected as route 1, and the other side is selected as route 2. Waypoints are taken on route 1 at set distance intervals. Assuming that any waypoint is M, the point closest to point M on route 2 is calculated and set as N. Then the center point of M and N is calculated, and curve fitting is performed on all center points to obtain a smooth inspection route; and the minimum inspection height of the drone is set for the obtained inspection route.

7. The method for inspecting a cable channel by using a drone according to claim 1, characterized in that: During the inspection phase, the drone autonomously flies according to the planned route and captures real-time images, identifying potential hazards and conducting alarm analysis of the location and movement direction for each frame of the image, including: For each frame of the image, a target detection algorithm is used to identify potential targets in the image, including various types of construction machinery and vehicles; when the potential targets are identified, an alarm analysis of the position and movement direction is started.

8. The method for inspecting a cable channel by using a drone according to claim 7, characterized in that: The alarm analysis method of the location is as follows: For each frame of image, obtain the GPS coordinate information of the drone when the frame was taken, and calculate the GPS coordinate information of each pixel in the frame according to the parameters such as the gimbal attitude angle and the focal length of the camera when shooting; If the GPS of a pixel is within the set channel alarm range, it is marked as 1, otherwise it is marked as 0, and the alarm binary image of the frame image is obtained; For each identified hidden danger target, a rectangular box is used to represent its position in the image. The pixel value in the rectangular box is marked as 1, otherwise it is marked as 0, and the target binary map of the image is obtained; Perform a union operation on the alarm binary image and the target binary image, and output an intersection mask image; If there is a point with a value of 1 in the intersection mask map, there is a hidden danger target within the warning range, and an alarm is issued.

9. The method for inspecting a cable channel by using a drone according to claim 7, characterized in that: After the position analysis, the alarm analysis of the moving direction is performed for the hidden danger targets that are not within the alarm range. The specific analysis steps are as follows: For any hidden danger target, track the position change of the hidden danger target in continuous image frames to obtain the pixel distance moved by the hidden danger target in the image. According to the time interval between images and the pixel distance moved, the moving speed of the hidden danger relative to the UAV can be calculated. The moving speed of the drone body is obtained through the flight control system, and the moving speed of the hidden danger relative to the ground is calculated based on the moving speed of the drone body and the moving speed of the hidden danger relative to the drone; When the moving speed of the hidden danger relative to the ground is 0, the hidden danger target is judged to be in a stationary state, otherwise it is in a moving state; For hidden danger targets in a moving state, continue to calculate their components in the direction orthogonal to the channel along the UAV. If the component is negative, it is determined that the moving direction is away from the channel. If the component is positive, it is determined that the moving direction is close to the channel, and an alarm is issued. If the component is zero, it is determined that the hidden danger is stationary at this time and is not in the alarm area.

10. A drone inspection system for cable channels, using the method according to any one of claims 1 to 9, characterized in that: The system comprises: The channel modeling module is used to use drones to inspect the cable channel under construction during the channel construction phase, and calculate and model the geographical location of the cable channel based on the language prompt word encoding module and closed operations; The route planning module is used to plan the cable channel inspection route based on the established cable channel model; The recognition and analysis module is used for the UAV to fly autonomously according to the planned route and take real-time images, identify potential danger targets for each frame of the image, and conduct alarm analysis on the position and movement direction.

11. A terminal comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images