Unmanned aerial vehicle autonomous electric power facility inspection method based on three-dimensional semantic driving

Abstract

An unmanned aerial vehicle autonomous electric power facility inspection method based on three-dimensional semantic driving relates to the technical field of unmanned aerial vehicle cruise control, and comprises the following steps: constructing a three-dimensional semantic representation containing key components under a unified world coordinate system, and generating a three-dimensional semantic point cloud with component category labels in real time in combination with airborne visual perception; and the inspection requirements in the operation and maintenance regulation are analyzed into an ordered component-level task sequence, and on the basis, the unmanned aerial vehicle autonomously plans an observation pose and a local flight path according to the three-dimensional semantic distribution of the target component, so that the adaptive approaching and coverage of the key component are realized. According to the invention, operation and maintenance semantics can be precisely followed, fine-grained sensing of key parts is realized, an inspection path is enabled to be close to a concern point, environment adaptability is realized, the inspection efficiency is effectively improved, the cost is reduced, and the problems that current electric power facility unmanned aerial vehicle inspection depends on a preset mode, the operation and maintenance semantics are difficult to meet, and key part sensing and adaptability are lacked are solved. And the inspection tour effect is not good enough.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle cruise control, and particularly relates to a three-dimensional semantic driving-based unmanned aerial vehicle autonomous power facility inspection method. BACKGROUND

[0002] In the current large-scale unmanned aerial vehicle inspection of power facilities, there is an excessive dependence on artificial preset routes and simple geometric constraints. This traditional mode has many drawbacks and cannot accurately reflect the semantic requirements in the operation and maintenance procedures, such as clearly checking the key components and specifying the encryption inspection method under abnormal conditions. At the same time, in the inspection process, there is a lack of fine-grained perception ability and the ability to adaptively adjust the inspection strategy according to the actual situation for key parts such as tower body, insulator string, fittings and blades, which directly leads to the disconnection between the inspection path and the actual points of interest, uneven inspection coverage, and lack of sensitivity to environmental changes.

[0003] Therefore, a three-dimensional semantic driving-based unmanned aerial vehicle autonomous power facility inspection method is needed to solve the above problems. SUMMARY

[0004] In view of the problems in the related art that the current large-scale unmanned aerial vehicle inspection of power facilities excessively depends on artificial preset routes and simple geometric constraints, cannot accurately reflect the semantic requirements in the operation and maintenance procedures, lacks fine-grained perception and adaptive adjustment ability for key parts, and causes disconnection of the inspection path, uneven coverage and lack of sensitivity to environmental changes, the present application provides a three-dimensional semantic driving-based unmanned aerial vehicle autonomous power facility inspection method.

[0005] The present application provides a three-dimensional semantic driving-based unmanned aerial vehicle autonomous power facility inspection method, comprising the following steps:

[0006] S1, a world coordinate system is established, and the power facility and the unmanned aerial vehicle are mapped to the world coordinate system to obtain an initial global facility three-dimensional semantic point cloud and an initial unmanned aerial vehicle pose ; the initial global facility three-dimensional semantic point cloud includes an initial global facility three-dimensional point cloud and an initial global semantic label vector ; the unmanned aerial vehicle is equipped with an onboard camera;

[0007] S2, after preprocessing a set of inspection task texts , the power component-related phrases are extracted and aligned with a power component category set to obtain a component category sequence space , and then a component-level semantic inspection sequence is generated according to the category sequence space .​ ;

[0008] S3. Based on the color RGB image acquired by the airborne camera at time t, obtain the pixel-level segmentation result of the target component through an open vocabulary visual perception network, and based on the depth map acquired by the airborne camera at time t... Camera intrinsic parameter matrix and the current pose of the drone After back-projecting the pixel-level segmentation results of the target component into three dimensions, the results are compared with the three-dimensional semantic point cloud of the global facility at the previous time step. The fusion process yields the current-time global facility 3D semantic point cloud. ;

[0009] S4, the first Category of electrical components awaiting inspection As the current target component category, then from the global facility 3D semantic point cloud Extract the 3D point set of the current target component category to generate representative waypoints for the current target component category. According to the current location of the drone and representative waypoints By using obstacle sets The constructed local path planning function yields a sequence of local control instructions. ;

[0010] S5. Based on the current pose of the drone Local control command sequence The control steps are executed step by step to obtain the new position and attitude of the UAV at the next moment. And update the global facility 3D semantic point cloud. Then in the new moment Filter and generate the 3D point set of the current target component And based on the three-dimensional point set of the UAV and the current target component. The closest distance between Construct a semantic inspection sequence at the component level to determine the coverage state indicator. Have all target components been inspected and completed?

[0011] Specifically, step S1 includes the following steps:

[0012] S11. Establish a world coordinate system and map the geometric point cloud data of the power facilities onto the world coordinate system to obtain a global 3D point cloud of the facilities. and a global facility 3D point cloud Initialize the corresponding initial global semantic label vector The initial three-dimensional semantic point cloud of the facility was obtained; Indicates the number of 3D points of the initial facility; Represents the set of electrical component categories; t represents time.

[0013] S12, Based on the initial position of the UAV With the initial rotation matrix Constructing the initial pose of the UAV in the world coordinate system .

[0014] Specifically, step S2 includes the following steps:

[0015] S21, Collection of inspection task texts Preprocessing yields a standardized collection of inspection task texts. ;

[0016] S22. Extract the inspection task text set Phrases related to electrical components and their association with categories of electrical components Alignment yields component category sequence space ;

[0017] S23, Component category sequence space All component categories are merged and deduplicated, and the order of different components is determined according to the explicit order words and implicit logical ordering of the operation and maintenance procedures to obtain the component-level semantic inspection sequence. .

[0018] Specifically, step S3 includes the following steps:

[0019] S31. Based on a frame of RGB image captured by the airborne camera at time t Inputting an open-vocabulary visual perception network yields pixel-level segmentation results of target components within the current field of view. ,in This represents the number of target components detected in the current frame. The component category label for the i-th target component; This is the semantic mask for the i-th target component;

[0020] S32, Traverse the pixel-level segmentation results All mask pixels For satisfying Mask pixels, based on the depth map acquired by the airborne camera at time t. Camera intrinsic parameter matrix and the current pose of the drone By backprojecting it in three dimensions, we obtain three-dimensional points in the world coordinate system, thus obtaining a set of three-dimensional points belonging to the i-th target component. ;

[0021] S33, All three-dimensional points of each target component at time t The incremental three-dimensional point cloud matrix of the current moment is constructed in column stacking , and the corresponding incremental semantic label vector is constructed , and the two are taken as the incremental three-dimensional semantic point cloud ;

[0022] S34, the incremental three-dimensional semantic point cloud is spliced and updated with the global facility three-dimensional semantic point cloud of the last moment to obtain the global facility three-dimensional semantic point cloud of the current moment.

[0023] Specifically, step S4 specifically includes the following steps:

[0024] S41, the first power component category to be inspected is taken as the current target component category, and then all points with the component category label equal to the current target component category are filtered out from the global facility three-dimensional semantic point cloud to generate the current target component three-dimensional point set ; l = 1, 2,..., L; L is the total number of target components;

[0025] S42, the point cloud geometric center of the current target component three-dimensional point set which is not empty is selected as the representative navigation point of the current target component ;

[0026] S43, according to the current position of the unmanned aerial vehicle and the representative navigation point of the current target component , a local path planning function based on the obstacle set of the global facility three-dimensional semantic point cloud is constructed to obtain a local control instruction sequence .

[0027] Specifically, step S5 specifically includes the following steps:

[0028] S51, according to the current pose of the unmanned aerial vehicle, the local control instruction sequence , and the control step h, the flight control is executed based on the flight control update function to obtain the new moment pose of the unmanned aerial vehicle , and the global facility three-dimensional semantic point cloud is updated through step S3; is the new moment; h is the control step;

[0029] S52, at each new moment , according to the global facility three-dimensional semantic point cloud ​Filter out all component category labels that are equal to the current target component category. The points generate the current component's 3D point set. And based on the drone's current location Calculate the 3D point set of the UAV and the current target component. The closest distance between ;

[0030] S53, Based on the nearest distance Construct coverage status indicator For the current target component category Perform semantic coverage determination of components, if Then the current target component category is considered to be... The semantic coverage requirement has been met, and the current target component category will be... Mark as completed inspection, then select the current target component category. Switch to component-level semantic inspection sequence The next target component category Proceed to step S4 until the component-level semantic inspection sequence is completed. All elements in the process are marked as having been inspected; if This indicates the current target component category. The semantic coverage has not yet been satisfied; the goal is to maintain [the target]. Proceed to step S3 to continue execution.

[0031] Specifically, the open vocabulary visual perception network described in step S31 can employ an existing open vocabulary detection and segmentation framework, the core structure of which is an image encoder plus a text encoder plus a cross-modal matching localization head, with the input being the current frame image. and the set of component categories The generated text prompts will be displayed on the network. The encoding is used to embed text and perform similarity matching with image features to locate the target element in the image. The corresponding region outputs the number of target components. and its masks and categories As the current frame image Pixel-level segmentation results of target components within the wide and high field of view.

[0032] Specifically, the local path planning function in step S43 first constructs a three-dimensional occupation grid map or Euclidean distance field map in the world coordinate system, and then uses the current UAV position... For the starting point and the destination waypoint The endpoint is based on the set of obstacles. Any feasible planning algorithm of A*, D Lite*, RRT on the map is used to obtain a collision-free discrete path, and finally the discrete path is smoothed and processed for dynamic feasibility to obtain a trajectory, and the trajectory is discretized into a control instruction sequence according to the flight control interface Output.

[0033] Specifically, in step S52, the current position of the unmanned aerial vehicle is determined according to the current position of the unmanned aerial vehicle The nearest distance between the unmanned aerial vehicle and the current target component three-dimensional point set The nearest distance between the unmanned aerial vehicle and the current target component three-dimensional point set As shown in the following formula:

[0034] ,

[0035] Wherein is the Euclidean norm function; Indicates the number of points related to the current target component category at the current time Indicates the number of points related to the current target component category at the current time Indicates the number of points related to the current target component category at the current time Indicates the point in the current target component three-dimensional point set Indicates the point in the current target component three-dimensional point set

[0036] Specifically, the value of the coverage state indication quantity in step S53 includes: if there is a time Make And , then the coverage state indication quantity Take 1; otherwise, the coverage state indication quantity Take 0; wherein, The distance threshold value for determining whether the semantic coverage meets the requirement.

[0037] The unmanned aerial vehicle autonomous power facility inspection method based on three-dimensional semantic driving provided by the application constructs a three-dimensional semantic representation containing key components under a unified world coordinate system, generates a three-dimensional semantic point cloud with component category labels in real time in combination with airborne visual perception, and analyzes the inspection requirements in the operation and maintenance regulations into an ordered component-level task sequence, on the basis of which the unmanned aerial vehicle autonomously plans observation pose and local flight path according to the three-dimensional semantic distribution of target components, realizes adaptive approach and coverage of key components, and can accurately follow operation and maintenance semantics, realize fine-grained perception of key parts, make the inspection path fit the points of interest, and have environmental adaptability, so as to realize unmanned aerial vehicle autonomous inspection of large-scale power facilities driven by three-dimensional semantic information as the core, and effectively improve the inspection efficiency, reduce the cost, and solve the problem that current unmanned aerial vehicle inspection of power facilities depends on a preset mode, is difficult to meet the operation and maintenance semantics, lacks key part perception and adaptability, and leads to poor inspection effect.

[0038] ​Further, the application proposes a three-dimensional semantic point cloud modeling method for power facilities: the tower body, insulator string, fittings, blades and other components are uniformly mapped into the world coordinate system through open vocabulary visual perception + RGB-D back projection, forming a three-dimensional semantic point cloud with geometric + component category labels, instead of the traditional model with only geometric shapes, which can have high-precision fine-grained perception capability for key parts such as tower body, insulator string, fittings and blades, and can sharply capture subtle defects and changes of these parts, such as cracks in insulator string, rust in fittings, surface damage in blades, etc., to timely discover potential safety hazards and nip fault hazards in the bud, effectively ensuring the safe and stable operation of power facilities.

[0039] Further, the application converts the textual description of the operation and maintenance procedures into executable component-level inspection sequences: through a semantic analysis function, the natural language requirements of what to inspect first and what to inspect later are mapped into an ordered component category sequence, and the semantic task chain is used to directly drive subsequent perception, planning and execution, instead of manually configuring fixed flight routes, which can deeply understand and accurately execute the semantic instructions in the operation and maintenance procedures, clearly check the key components, and automatically adjust the inspection method according to different abnormal conditions, improving the professionalism and effectiveness of the inspection work.

[0040] Further, the application also has an autonomous pose planning and coverage determination mechanism based on three-dimensional semantic point cloud: around the current target component, corresponding point sets are extracted from the semantic point cloud and three-dimensional representative navigation points are calculated, local control instruction sequences are generated in combination with obstacle constraints, and whether the component has been inspected is automatically determined by the minimum distance from the unmanned aerial vehicle position to the component point set + coverage threshold, realizing adaptive inspection path and coverage control directly driven by semantic targets, which can autonomously plan an inspection path that closely matches the actual points of interest, avoiding the problem of disconnection between the path and the points of interest in the traditional mode, ensuring comprehensive and uniform inspection coverage without missing any key areas, and improving the integrity and reliability of the inspection work.

[0041] Further, the application reduces the workload of manual intervention and preset flight routes, and optimizes the inspection path through autonomous planning and intelligent decision-making, greatly shortening the inspection time and improving the inspection efficiency; moreover, accurate inspection can discover and handle problems in advance, avoiding greater losses caused by fault expansion, and effectively reducing the operation and maintenance cost of power facilities in the long run. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0043] Figure 1 This is a schematic diagram of an autonomous power facility inspection method based on three-dimensional semantic driving, provided as an embodiment of the present invention. Detailed Implementation

[0044] The invention will be explained in detail through the following embodiments. The purpose of this invention is to protect all technical improvements within its scope. In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "multiple" means two or more, unless otherwise explicitly specified.

[0045] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0046] Example 1

[0047] refer to Figure 1 This embodiment provides a method for autonomous inspection of power facilities by unmanned aerial vehicles (UAVs) based on three-dimensional semantic driving, including the following steps:

[0048] S1. Establish a world coordinate system, map the power facilities and drones onto the world coordinate system, and obtain the initial global 3D semantic point cloud of the facilities. and the initial pose of the drone The initial global facility three-dimensional semantic point cloud Including the initial global facility 3D point cloud and the initial global semantic label vector The drone is equipped with an onboard camera.

[0049] S11. Establish a world coordinate system and map the geometric point cloud data of the power facilities onto the world coordinate system to obtain a global 3D point cloud of the facilities. and a global facility 3D point cloud Initialize the corresponding initial global semantic label vector The initial three-dimensional semantic point cloud of the facility was obtained; Indicates the number of 3D points of the initial facility; Represents the set of electrical component categories; t represents time.

[0050] First, select a unified global coordinate system for the area where the power facilities are located. For example, take the center of the transmission tower base or the bottom of the wind turbine tower as the origin, and vertically upwards as... The axis, the horizontal direction is defined as shaft and axis; then the collected geometric point cloud data of various power facilities (whether from different surveying equipment or different format models) are mapped into the same world coordinate system through field calibration and coordinate conversion, taking t=0 as the initial time, to obtain the initial global facility three-dimensional point cloud , abbreviated as ; wherein represents the initial facility three-dimensional point quantity, abbreviated as ;

[0051] At time t, each column of represents the coordinates of a three-dimensional point on the power facility (such as a power transmission tower, a wind turbine tower, a blade, etc.) in the world coordinate system, which is a three-dimensional vector, i.e. it consists of three components, which can usually be represented as , wherein respectively represent the coordinate values of the x-axis, y-axis, and z-axis of the world coordinate system at time t; n=1,..., N t ; represents the facility three-dimensional point quantity;

[0052] That is, at the initial time t=0, each column of represents the coordinates of a three-dimensional point on the power facility (such as a power transmission tower, a wind turbine tower, a blade, etc.) in the world coordinate system;

[0053] Specifically, the geometric point cloud data of various power facilities can be obtained by laser radar scanning point cloud and BIM / CAD model sampling point cloud;

[0054] In order to bind the geometric points with the power component categories in the subsequent process, an initial global semantic label vector is initialized for the initial global facility three-dimensional point cloud , abbreviated as ; wherein represents the component category label corresponding to the th geometric point , which takes a value from the power component category set .

[0055] The operation and maintenance side pre-defines a set of power component category set that needs to be paid attention to:

[0056]

[0057] wherein is a component category label (such as tower body insulator string hardware blade root, etc.), and is the total number of categories. In this step, the can be uniformly initialized as unannotated or a certain general category placeholder;

[0058] S12, based on the initial position of the UAV and the initial rotation matrix to construct the initial pose of the UAV in the world coordinate system :

[0059] the current pose of the UAV in the world coordinate system at time t is represented by the following homogeneous transformation matrix:

[0060] ,

[0061] wherein, is the current rotation matrix, representing the rotation matrix of the UAV at time t, is the current position of the UAV, representing the three-dimensional position of the UAV in the world coordinate system at time t, is a zero row vector;

[0062] the current position of the UAV and the current rotation matrix are directly output by the UAV navigation / positioning module (such as SLAM) at time t The current pose is uniformly written as a homogeneous matrix Therefore, can also be regarded as the translation term extracted by Meanwhile, is the initial pose constructed by the initial at time t is updated over time by the positioning module, so the two are instances of the same pose at different times, and there is a clear time evolution relationship.

[0063] Therefore, at time t = 0, based on the initial position of the UAV and the initial rotation matrix to construct the initial pose of the UAV in the world coordinate system :

[0064] ,

[0065] This matrix is used to describe the rigid transformation of the camera coordinate system / body coordinate system→world coordinate system.

[0066] The initial position and attitude of the UAV in the world coordinate system are recorded as the position vector and the rotation matrix ; wherein gives the three-dimensional coordinates of the take-off or inspection starting point of the UAV, ​Then the direction transformation from the world coordinate system to the UAV body coordinate system is described, both of which are used to construct the initial UAV homogeneous pose matrix . The variable is initialized to by step S1 and updated over time during the inspection process;

[0067] The UAV is equipped with an on-board camera, and the intrinsic parameters of the on-board camera are represented by , which is called the camera intrinsic matrix, which contains parameters such as focal length and principal point coordinates.

[0068] The on-board camera is an RGB-D camera;

[0069] An RGB-D camera is a sensor that can simultaneously acquire color images (RGB) and depth information (Depth). By combining traditional color cameras and depth sensors, it provides more rich three-dimensional scene information than single RGB cameras.

[0070] The camera intrinsic matrix does not perform numerical operations itself, but its relationship with the world coordinate system and the UAV pose is clearly defined: in the subsequent steps, the pixel coordinates will be normalized and mapped to the camera coordinate system, and then converted to the world coordinate system through the current pose matrix to obtain the three-dimensional coordinates in the three-dimensional semantic point cloud. By fixing in this step, it can ensure that the projection calculation at different time steps in the entire inspection process follows the same camera model, facilitating the continuous accumulation of semantic point clouds.

[0071] This step unifies large-scale power facilities, UAVs, and cameras under the same three-dimensional world coordinate system and initializes the core variables used in subsequent semantic point cloud construction and path planning.

[0072] S2, after preprocessing the inspection task text set , extract the power component related phrases and align them with the power component category set to obtain the component category sequence space , and then generate the component-level semantic inspection sequence according to the category sequence space ;

[0073] S21, after preprocessing the inspection task text set , obtain the standardized inspection task text set ;

[0074] The operation and maintenance unit gives a set of inspection task descriptions according to the regulations or on-site needs, which are uniformly recorded as the inspection task text set ​​:

[0075] wherein, is the number of text entries, each is a text entry (e.g., check insulator string first, then cross arm connecting bolt, and finally tower top lightning rod, etc.) described in natural language text, which can be viewed as a sequence defined on the string space , i.e. . These texts collectively describe what to check and the general order of the sequence in the current inspection task, etc.

[0076] First, the inspection task text set is preprocessed, including unified encoding format, removing irrelevant symbols, sentence segmentation and word segmentation, etc., to obtain text data with clearer structure, more standardized content, and convenient for further in-depth analysis and mining. After preprocessing, the standardized text data is updated to the inspection task text set ;

[0077] S22, extract power component-related phrases in the inspection task text set and align them with the power component category set to obtain the component category sequence space ;

[0078] On this basis, domain rules and large model reasoning are used to identify power component-related phrases in the text and align them with the power component category set . This process can be formalized as a semantic extraction function :

[0079]

[0080] Through this function, phrases such as insulator string, cross arm, tower top lightning rod, and blade leading edge appearing in each natural language text are identified, and they are mapped to specific component category labels using matching rules or semantic similarity judgments. ; denotes the component category sequence space with length , and is the total number of target components obtained after parsing.

[0081] S23, merge and remove duplicates of all component categories in the component category sequence space , and determine the order of different components according to the explicit order words and implicit logical sorting of the operation and maintenance procedures to obtain the component-level semantic inspection sequence ;

[0082] After the extraction and category matching of key component phrases, the component category sequence space The component class candidates obtained in all text entries are merged, de-duplicated, and sorted. Specifically, on the one hand, the order relationship between different components is determined according to the explicit order words (such as first, second, and last) given in the operation and maintenance regulations and the implicit logic (such as checking high-risk parts first and then checking regular components); on the other hand, the same component class repeatedly appears in multiple texts is merged, and finally an ordered component-level semantic inspection sequence is obtained :

[0083]

[0084] wherein, represents the i-th power component class to be inspected, is the length of the component-level semantic inspection sequence. The sequence can be regarded as the output result of , which not only retains the semantic requirements of what to inspect in the operation and maintenance regulations, but also gives the execution path of what order to inspect, providing a direct basis for subsequent filtering of semantic point clouds and planning observation poses according to the current component class .

[0085] This step converts the natural language task description of the operation and maintenance regulations into an ordered power component-level inspection list, and the subsequent steps are all according to this list to perceive and plan one component at a time.

[0086] S3, obtaining the pixel-level segmentation result of the target component through the open vocabulary visual perception network according to the color RGB image collected by the airborne camera at time t, and obtaining the three-dimensional semantic point cloud of the global facility at time t by three-dimensionally back-projecting all mask pixels meeting the requirements in the pixel-level segmentation result of the target component and fusing the three-dimensional semantic point cloud of the global facility at the previous time , the camera intrinsic matrix , and the current pose of the unmanned aerial vehicle . ;

[0087] The unmanned aerial vehicle collects a frame of color RGB image and a corresponding depth map at time t by the airborne camera, wherein are the image height and width, respectively, represents the RGB value (channel dimension is 3) of the pixel at the pixel coordinate , represents the depth distance from the pixel to the camera optical center.

[0088] It can be understood that the three-dimensional semantic point cloud of the global facility includes ​​, both of which are constantly updated according to time t; if time t is taken as the current time, the current global facility three-dimensional semantic point cloud is , and the semantic point cloud constructed or updated at the previous time (time t-1) is taken as the previous global facility three-dimensional semantic point cloud, denoted as:

[0089] , wherein represents the number of three-dimensional points of the facility at the previous time, each column of the matrix is a three-dimensional point coordinate in the world coordinate system, is a component category label corresponding to the three-dimensional point, and takes a value in the power component category set At , the current global facility three-dimensional semantic point cloud is the initial global facility three-dimensional point cloud and the initial global semantic label vector , and there is no previous global facility three-dimensional semantic point cloud;

[0090] S31, according to a frame of RGB image collected by the airborne camera at time t , the pixel-level segmentation result of the target component in the current field of view is obtained by inputting the open-vocabulary visual perception network , which can be formalized as:

[0091]

[0092] , wherein is the open-vocabulary visual perception network; is the number of target components detected in the current frame; is the component category label of the i-th target component, indicating which type of power component (such as an insulator string cross-arm tower body) the target component belongs to; is the semantic mask of the i-th target component, which is a binary matrix, indicates that the mask pixel belongs to the target component, indicates that the mask pixel does not belong to the target component; H is the image height; W is the image width;

[0093] Through this open-vocabulary detection and segmentation method, the region related to the power component category set can be directly extracted from the image without predefining a fixed target set.

[0094] The open vocabulary visual perception network can employ existing open vocabulary detection and segmentation frameworks (e.g., a combination of an open vocabulary detector and a general segmenter, or a standalone open vocabulary segmentation network). Its core structure can be summarized as an image encoder + a text encoder + a cross-modal matching localization head (which can be connected to a segmentation head), with the input being the current frame image. and the set of component categories The generated text prompts (such as category names / synonyms) will be used by the network. The encoding is used to embed text and perform similarity matching with image features to locate the target element in the image. The corresponding region outputs the number of target components. and its masks and categories As the current frame image The pixel-level segmentation results of the target components within the wide and high field of view.

[0095] Compared to conventional closed-vocabulary networks, this type of open-vocabulary method does not require retraining the classifier head for each new component category; it only needs to update / expand the classifier. The text prompts can be used to detect the problem, making it more suitable for the needs of power inspection that require scalable categories, long tails, and cross-scenario generalization.

[0096] S32, Traverse the pixel-level segmentation results All mask pixels For satisfying Mask pixels, based on the depth map acquired by the airborne camera at time t. Camera intrinsic parameter matrix and the current pose of the drone By backprojecting it in three dimensions, we obtain three-dimensional points in the world coordinate system, thus obtaining a set of three-dimensional points belonging to the i-th target component. As shown in the following formula:

[0097]

[0098]

[0099] in, For the homogeneous coordinates of the mask pixels, It is the inverse of the camera intrinsic parameter matrix. For mask pixels The corresponding homogeneous world coordinate vector, This indicates that the first three components of the vector are used as the three-dimensional coordinates.

[0100] S33, All three-dimensional points of each target component at time t The points are stacked column-wise to form the incremental 3D point cloud matrix at the current time. And construct its corresponding incremental semantic label vector. , as the incremental three-dimensional semantic point cloud ;

[0101] After the inverse projection of the mask pixels to the world coordinates is completed, all three-dimensional points of each target component (the i-th target component) at time t are stacked column by column to form an incremental three-dimensional point cloud matrix at the current time , wherein is the number of newly added three-dimensional points in the current frame.

[0102] The incremental three-dimensional point cloud matrix The corresponding incremental semantic label vector is , wherein for all points belonging to the i-th target component, the label element is , since , so ;

[0103] In another possible implementation, repeated or noisy points can also be simply filtered or down-sampled in combination with depth values, normal vectors or confidence, but the above representation form is not changed.

[0104] S34, the incremental three-dimensional semantic point cloud is spliced and updated with the global facility three-dimensional semantic point cloud at the previous time to obtain the global facility three-dimensional semantic point cloud at the current time.

[0105] The incremental three-dimensional point cloud matrix at the current time is spliced and updated with the global facility three-dimensional point cloud at the previous time to obtain the global facility three-dimensional point cloud at the current time, the incremental semantic label vector is spliced and updated with the global semantic label vector at the previous time to obtain the global semantic label vector at the current time.

[0106]

[0107]

[0108] , wherein the symbol represents a column splicing operation; is the total number of facility points after updating, i.e., the number of facility three-dimensional points at the current time; is the number of facility three-dimensional points at the previous time.

[0109] By accumulating data frame by frame, a continuously enriched 3D semantic map is formed in the world coordinate system. This map preserves the geometric shape and records the component category corresponding to each point, providing a basis for subsequent analysis based on the current target component. It provides a data foundation for selecting corresponding regions and generating inspection poses in point clouds.

[0110] This step utilizes an onboard RGB-D camera from a drone to perform open-vocabulary visual perception of power facilities, and back-projects pixels belonging to various component categories onto the world coordinate system to construct and update a 3D point cloud with semantic labels. This provides a foundation for subsequent semantic-driven pose planning and coverage determination. In the following steps, a component-level semantic inspection sequence will be used. With the goal of using a global facility 3D semantic point cloud Filtering and current component category The corresponding point set is used to generate a 3D semantically driven observation pose and local flight path; then, the semantic coverage is calculated using this point cloud information and the completion of component inspection is determined.

[0111] S4, the first Category of electrical components awaiting inspection As the current target component category, then from the global facility 3D semantic point cloud Extract the 3D point set of the current target component category to generate representative waypoints for the current target component category. According to the current location of the drone and representative waypoints By using obstacle sets The constructed local path planning function yields a sequence of local control instructions. ;

[0112] Step S2 generates a component-level semantic inspection sequence. ,in Indicates the first The categories of electrical components to be inspected (e.g., insulator strings, tower top lightning rods, etc.), l=1, 2, ..., L; This represents the total number of target components. This step uses a fixed... To achieve this, we select the corresponding semantic regions from the point cloud and generate inspection poses and local paths pointing to those semantic regions.

[0113] Current moment global facility 3D point cloud Each column This represents the coordinates of a three-dimensional point on a power facility (such as a transmission tower, wind turbine tower, or blade) in the world coordinate system. These coordinates are a three-dimensional vector, meaning they consist of three components, and can typically be represented as: ,in Let x, y, and z represent the coordinates of the x-axis, y-axis, and z-axis in the world coordinate system at time t, respectively; n = 1, ..., N. t ; Indicates the number of three-dimensional points of the facility;

[0114] Based on the existing 3D semantic point cloud, focusing on the current target component category It automatically selects its three-dimensional representative area / point and generates a local control command sequence for the UAV to fly from its current position to the vicinity of the area under obstacle avoidance constraints, providing trajectory input for the subsequent autonomous execution and coverage determination of S5.

[0115] S41, the first Category of electrical components awaiting inspection As the current target component category, then from the global facility 3D semantic point cloud Filter out all points whose component category labels are equal to the current target component category to generate the current target component's 3D point set. l = 1, 2, ..., L; L is the total number of target components;

[0116]

[0117] in, For the current target component category In the current global facility 3D semantic point cloud The number of points in; if This indicates that there is currently no effective observation of the component from the current perspective, and further observation can be achieved in subsequent steps by continuing to move and supplement the perception. Gradually enriched; if Then it can be assumed that the component already has a certain geometric and semantic description at the current moment, and the current target component's three-dimensional point set It is a three-dimensional point set in the world coordinate system, which represents the current component and is suitable for generating inspection poses based on it.

[0118] S42. Select a non-empty 3D point set of the current target component. The geometric center of the point cloud serves as the representative waypoint of the current target component. ;

[0119] when At that time, that is, the current three-dimensional point set of the component Not empty; to simplify the process at the planning level, [the value will be]... The geometric information of all 3D points in the cloud is compressed into a single representative waypoint, using the geometric center of the point cloud:

[0120]

[0121] Among them, the representative waypoint of the current component represents the three-dimensional representative position (waypoint) of the current component at the current time t, taking into account the overall distribution characteristics of the point cloud.

[0122] In another possible implementation, a safety offset or perspective constraint can also be added near the geometric center of the point, but the symbol level is uniformly used represents the representative position of the component to be reached, which is convenient for subsequent comparison with the position of the unmanned aerial vehicle together input to the path planning function.

[0123] S43, according to the current position of the unmanned aerial vehicle and the representative waypoint of the current target component , the local path planning function constructed based on the global facility three-dimensional semantic point cloud of the obstacle set obtains a local control instruction sequence ; as shown in the following formula:

[0124] ,

[0125] wherein, is the local control instruction sequence obtained by planning; is the hth control vector, which includes expected speed, attitude or position increment, etc., and the dimension is related to the specific flight control interface; is the planning step number; h is the control step, h = 1, 2,..., H; the current position of the unmanned aerial vehicle as the starting position of the current path planning;

[0126] According to the point distribution and label in the global facility three-dimensional semantic point cloud , the tower body, ground and cross arm and other regions that are not allowed to be crossed can be clustered into a plurality of obstacle bodies in advance, which are uniformly denoted as an obstacle set :

[0127]

[0128] wherein is the number of obstacles at the current time t, and each represents an obstacle region (which can be represented by a point cloud envelope or a simplified geometric body) in the world coordinate system; the set is used as an obstacle avoidance constraint condition in the planning process.

[0129] is the local path planning function, and the function needs to meet the following requirements in the planning process: first, the trajectory continuously approaches and finally reaches the vicinity of the target waypoint from the starting point ; second, avoid The obstacles, such as the tower and the ground, are shown in the figure. The safety distance and flight altitude constraints are met. Third, under the premise of meeting the safety constraints, the path length, number of turns, or energy consumption are optimized as much as possible.

[0130] Local path planning function The implementation can be completed according to the process of mapping—searching—trajectory generation—control output. Specifically, it starts with the global semantic point cloud. Cluster the impassable regions and expand the safety distance to obtain the obstacle set. And construct a 3D occupation grid map, Euclidean distance field map (ESDF), or navigation map in the world coordinate system; then, based on the current drone position For the starting point and the destination waypoint With the destination as the endpoint, a collision-free discrete path is obtained on the map using any feasible planning algorithm such as A*, D Lite*, or RRT. Finally, the discrete path is smoothed and subjected to dynamic feasibility processing (such as B-spline / minimum jerk) to obtain the trajectory, which is then discretized into a sequence of control commands via the flight control interface. The output, implemented as described above, belongs to A typical implementation can be achieved by selecting an equivalent planner based on computing power and platform.

[0131] The planning results are in the form of a sequence of control commands. The output will be in the form of [output format], and in subsequent steps, the drone will follow [the instructions / methods]. Control commands are executed sequentially, and the global semantic point cloud is continuously updated using the S3 mechanism during execution. Further calculate the component coverage distance and its relationship with the preset threshold The relationship between these parameters determines whether the current component inspection is complete; therefore, this step defines... , , and The symbols will be used directly in S5 and will not be redefined.

[0132] S5. Based on the current pose of the drone Local control command sequence The control steps are executed step by step to obtain the new position and attitude of the UAV at the next moment. And update the global facility 3D semantic point cloud. Then in the new moment Filter and generate the 3D point set of the current target component And based on the three-dimensional point set of the UAV and the current target component. The closest distance between Construct a semantic inspection sequence at the component level to determine the coverage state indicator. Have all target components been inspected and completed?

[0133] S51, based on the current position of the drone Local control command sequence The control step h is based on the flight control update function to execute flight control step by step to obtain the new pose of the UAV at the new moment. And update the global facility 3D semantic point cloud through step S3. ; The new time step; h represents the control step;

[0134] First, the drone follows the sequence of local control commands. Flight control is executed incrementally. To characterize the relationship between control and pose, a flight control update function can be defined:

[0135] ,

[0136] in Indicates the current pose of the drone Apply control vector The pose obtained at the next moment. For each control step Get the corresponding new moment ( The new time of the UAV is obtained by recursion through the flight control update function. Corresponding new drone pose at the new time And extract the new time position of the drone from it. .

[0137] In every new moment The UAV utilizes the open-vocabulary visual perception and 3D back-projection mechanism in step S3 to re-perceive and project the current field of view, thus creating a 3D semantic point cloud of the global facilities. Replace t with This will update the semantic point cloud. and ,in The updated points allow the map to continuously accumulate and be corrected during flight.

[0138] S52, in every new moment Based on the global facility 3D semantic point cloud Filter out all component category labels that are equal to the current target component category. The points generate the current component's 3D point set. And based on the drone's current location Calculate the 3D point set of the UAV and the current target component. The closest distance between ;

[0139] Referring to step S41, at each new time instant , the updated global facility 3D semantic point cloud is referred to extract the 3D point set of the current target component class as the current target component 3D point set :

[0140]

[0141] wherein denotes the current time instant and the number of points related to the current target component class ;

[0142] Subsequently, the closest distance between the UAV and the current component 3D point set is calculated with reference to the current UAV position :

[0143] ,

[0144] wherein is the Euclidean norm function; , used to measure the length of a 3D vector; denotes the points in the current target component 3D point set . This distance can characterize the spatial proximity of the UAV to the closest visible point of the target component.

[0145] S53, according to the closest distance , a coverage state indicator is constructed to make a component semantic coverage determination for the current target component class . If , it is considered that the semantic coverage requirement of the current target component class has been met, and the current target component class is marked as having completed the inspection, and then the current target component class is switched to the next target component class in the component-level semantic inspection sequence , and jumps to step S4, until all elements in the component-level semantic inspection sequence are marked as having completed the inspection; if , it indicates that the semantic coverage of the current target component class has not been met, at this time the system will not switch to , but remains the target as , and continues to execute the perception update (S3) - planning (S4) - execution (S5) closed loop; ​

[0146] If there are still after the execution of the current control instruction sequence , trigger a new path planning, directly jump to step S4 to generate a new control instruction sequence, so as to continue to attempt to approach the component under the new starting point and the updated point cloud condition;

[0147] If there is a time , so that and , the coverage state indication quantity takes 1; otherwise, the coverage state indication quantity takes 0; as follows:

[0148]

[0149] , wherein is the distance threshold for determining whether the semantic coverage meets the requirements; is a scalar real number, representing the maximum distance allowed between the UAV and the target component region (for example, if the UAV is less than or equal to from the surface of the target component, it is considered that the component has been effectively covered);

[0150] After the execution of the current control instruction sequence , there are still It is emphasized that the checkpoint after one planning-execution cycle, if still not up to standard, then re-call S4 to generate a new with a new starting point (i.e. the current position / pose of the UAV after the execution of ), and continue to attempt to approach and cover the component under the updated point cloud condition (i.e. the accumulated through S3 during the execution process); , The termination condition is that all components in the component-level semantic inspection sequence corresponding to the coverage state vector; all components are 1, that is, each type of target component has met the coverage requirements.

[0151] Finally, the output includes the state coverage vector of the coverage state indication quantity of each type of power component to be inspected in the component-level semantic inspection sequence .

[0152] , wherein indicates that the component has met the semantic coverage requirements, indicates that it is not completed or needs to be re-planned.

[0153] ​The final three-dimensional semantic point cloud accumulated during the entire inspection process:

[0154] ,

[0155] in, This marks the end of the inspection. This represents the number of 3D points in the final point cloud, which can be used for defect identification and visualization in post-processing.

[0156] The set of pose trajectories of the drone throughout the inspection process:

[0157] ,

[0158] in These are a series of discrete location points during the inspection process. Corresponding to all components All were judged At that moment.

[0159] pose trajectory set Its function is to record the flight path of the drone during the inspection process, for purposes such as mission review, coverage visualization, mileage / energy consumption statistics, and safety auditing; where discrete location points refer to points within discrete time steps. Upsampled UAV position vector This refers to the three-dimensional position of the drone in the world coordinate system.

[0160] This embodiment provides a method for autonomous UAV inspection of power facilities based on 3D semantics. It constructs a 3D semantic representation of key components within a unified world coordinate system, and generates a 3D semantic point cloud with component category labels in real time using onboard visual perception. The inspection requirements in the operation and maintenance procedures are parsed into an ordered sequence of component-level tasks. Based on this, the UAV autonomously plans its observation pose and local flight path according to the 3D semantic distribution of the target components, achieving adaptive approach and coverage of key components. This embodiment accurately follows operation and maintenance semantics, achieves fine-grained perception of key parts, ensures the inspection path aligns with the points of interest, and possesses environmental adaptability. Therefore, it enables autonomous UAV inspection of large-scale power facilities driven by 3D semantic information, effectively improving inspection efficiency and reducing costs. It solves the problems of current power facility UAV inspections relying on preset modes, failing to meet operation and maintenance semantics, lacking key part perception and adaptability, resulting in poor inspection results.

[0161] Further, the embodiment proposes a three-dimensional semantic point cloud modeling method for power facilities: the tower body, insulator string, fittings, blades and other components are uniformly mapped into the world coordinate system through open vocabulary visual perception + RGB-D back projection, forming a three-dimensional semantic point cloud with geometric + component category labels, instead of the traditional model with only geometric shapes. This way, it can have high-precision fine-grained perception capability for key parts such as tower body, insulator string, fittings, blades, etc., and can sharply capture subtle defects and changes in these parts, such as cracks in insulator strings, rust in fittings, surface damage in blades, etc., to timely discover potential safety hazards and nip fault hazards in the bud, effectively ensuring the safe and stable operation of power facilities.

[0162] Further, the embodiment converts the textual description of the operation and maintenance procedures into executable component-level inspection sequences: through a semantic analysis function, it maps natural language requirements such as what to inspect first and what to inspect later into an ordered component category sequence, directly drives subsequent perception, planning and execution with semantic task chains, rather than manually configuring fixed flight routes, deeply understands and accurately executes semantic instructions in operation and maintenance procedures, clearly checks key components, automatically adjusts inspection methods according to different abnormal situations, and improves the professionalism and effectiveness of inspection work.

[0163] Further, the embodiment also has an autonomous pose planning and coverage determination mechanism based on three-dimensional semantic point cloud: around the current target component, it extracts the corresponding point set from the semantic point cloud and calculates the three-dimensional representative navigation point, generates a local control instruction sequence in combination with obstacle constraints, and automatically determines whether the component has been inspected by the minimum distance from the unmanned aerial vehicle position to the component point set + coverage threshold, realizes adaptive inspection path and coverage control directly driven by semantic targets, can autonomously plan an inspection path that closely matches the actual points of interest, avoids the problem of disconnection between the path and the points of interest in the traditional mode, ensures comprehensive and uniform inspection coverage without missing any key areas, and improves the integrity and reliability of the inspection work.

[0164] Further, since the workload of manual intervention and preset flight routes is reduced, and the inspection path is optimized through autonomous planning and intelligent decision-making, the inspection time is greatly shortened and the inspection efficiency is improved. Moreover, accurate inspection can discover and handle problems in advance, avoiding greater losses caused by fault expansion, effectively reducing the operation and maintenance cost of power facilities in the long run.

[0165] The specification and drawings are directed to one or more implementations of the methods, devices (systems), and computer program products described herein. Although the following exemplary implementations will be described with reference to a method, device (system), and computer program product embodied in hardware and / or software, it will be appreciated that equivalents and / or modifications of the implementations described herein can be made in hardware and / or software. For example, the implementations described herein can be implemented in hardware, software, or a combination of hardware and software. In addition, the implementations described herein can be implemented in a distributed manner, such as over the Internet, or in a stand-alone manner. Figure 1 one or more of the steps in a flow, and / or combinations of the steps in a flow. Figure 1 a device configured to perform the functions specified in one or more of the steps in a flow, and / or combinations of the steps in a flow.

[0166] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flow, or multiple flows, and / or the functions specified in one or more of the steps in a flow, or multiple flows. Figure 1 one or more of the steps in a flow, and / or combinations of the steps in a flow. Figure 1 a device configured to perform the functions specified in one or more of the steps in a flow, or multiple flows, and / or combinations of the steps in a flow, or multiple flows.

[0167] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow, or multiple flows, and / or the functions specified in one or more of the steps in a flow, or multiple flows. Figure 1 one or more of the steps in a flow, and / or combinations of the steps in a flow. Figure 1 a device configured to perform the functions specified in one or more of the steps in a flow, or multiple flows, and / or combinations of the steps in a flow, or multiple flows.

[0168] The present application is not limited to the details of the foregoing exemplary implementations and can be practiced with modification and alteration within the scope and spirit of the present application. Accordingly, the specification and drawings are to be regarded in an illustrative, rather than a restrictive, sense. The implementations are illustrative only and are not intended to be limiting on the scope of the present application as would be described by the language of the following claims.

Claims

1. A three-dimensional semantic driving based unmanned aerial vehicle autonomous power facility inspection method, characterized in that, The method comprises the following steps: S1, establish a world coordinate system, map the power facility and the unmanned aerial vehicle to the world coordinate system, and obtain an initial global facility three-dimensional semantic point cloud and an initial pose of the unmanned aerial vehicle ; the initial global facility three-dimensional semantic point cloud comprises an initial global facility three-dimensional point cloud and an initial global semantic label vector ; the unmanned aerial vehicle is provided with an onboard camera; S2, to the inspection task text set After preprocessing, extract the power component related phrases and match them with the power component category set Align to get component category sequence space Then according to the category sequence space Generate component-level semantic inspection sequence ; S3, obtaining a pixel-level segmentation result of the target component through an open-vocabulary visual perception network according to an image collected by the onboard camera at the time t, and obtaining a depth map of the target component according to the image collected by the onboard camera at the time t , a camera intrinsic matrix , and a current pose of the unmanned aerial vehicle , performing three-dimensional back-projection on all mask pixels meeting the requirements in the pixel-level segmentation result of the target component, and fusing the three-dimensional back-projection result and a global facility three-dimensional semantic point cloud at a previous time to obtain a global facility three-dimensional semantic point cloud at the current time ;​ S4, the first Category of electrical components awaiting inspection As the current target component category, then from the global facility 3D semantic point cloud Extract the 3D point set of the current target component category to generate representative waypoints for the current target component category. According to the current location of the drone and representative waypoints By using obstacle sets The constructed local path planning function yields a sequence of local control instructions. ; S5、according to the current pose of the UAV , a local control instruction sequence , and the UAV new time pose is obtained by gradually executing flight control in steps , and the global facility three-dimensional semantic point cloud is updated , then at the new time , the current target component three-dimensional point set is generated by screening , and the closest distance between the UAV and the current target component three-dimensional point set is calculated , the coverage state indication quantity is constructed to determine the component-level semantic inspection sequence whether all target components in the sequence are inspected.

2. The method of claim 1, wherein, The step S1 specifically comprises the following steps: S11. Establish a world coordinate system, map the geometric point cloud data of the power facility to the world coordinate system to obtain a global facility three-dimensional point cloud , and for the global facility three-dimensional point cloud Initialize the corresponding initial global semantic label vector Obtain an initial facility three-dimensional semantic point cloud; Indicates the number of initial facility three-dimensional points; Indicates the set of power component categories; t indicates the time; S12, determining the initial position of the UAV based on the initial position of the UAV with the initial rotation matrix constructing the initial pose of the UAV in the world coordinate system based on the initial position of the UAV .

3. The method of claim 1, wherein, The step S2 specifically comprises the following steps: S21, obtaining a set of inspection task texts obtaining a set of normalized inspection task texts by preprocessing ​ S22, extract the inspection task text set power component related phrases and classify them into a power component category set align to obtain a component category sequence space ; S23, merge, deduplicate all component categories, and determine the order of different components according to the explicit order words and implicit logic of the operation and maintenance procedures to obtain a component-level semantic inspection sequence merge, deduplicate all component categories, and determine the order of different components according to the explicit order words and implicit logic of the operation and maintenance procedures to obtain a component-level semantic inspection sequence .

4. The method of claim 1, wherein, The step S3 specifically comprises the following steps: S31, obtaining a frame of RGB image collected by the on-board camera at the time t inputting the open-vocabulary visual perception network to obtain a pixel-level segmentation result of the target component in the current field of view wherein is the number of target components detected in the current frame; is the component class label of the i-th target component; is the semantic mask of the i-th target component; S32, traversing the pixel-level segmentation result all the mask pixels , for the mask pixel satisfying , according to the depth map collected by the on-board camera at the moment t , the camera intrinsic matrix , and the current pose of the unmanned aerial vehicle , three-dimensionally back-projecting the same to obtain a three-dimensional point in the world coordinate system, and obtaining a group of three-dimensional points belonging to the i th target component ; S33, all three-dimensional points of each target member at time t are stacked by column to build an incremental three-dimensional point cloud matrix of the current time , and build its corresponding incremental semantic label vector , and both are taken as an incremental three-dimensional semantic point cloud ; S34, incrementing the three-dimensional semantic point cloud with the global facility three-dimensional semantic point cloud of the previous time to obtain the global facility three-dimensional semantic point cloud of the current time .

5. The method of claim 1, wherein, The step S4 specifically comprises the following steps: S41, the first Category of electrical components awaiting inspection As the current target component category, then from the global facility 3D semantic point cloud Filter out all points whose component category labels are equal to the current target component category to generate the current target component's 3D point set. l = 1, 2, ..., L; L is the total number of target components; S42, selecting the point cloud geometric center of the non-empty current target component three-dimensional point set as a representative waypoint of the current target component ;​ S43, according to the current position of the unmanned aerial vehicle and the representative waypoint of the current target member The local path planning function constructed by the global facility three-dimensional semantic point cloud based on the obstacle set The local control instruction sequence is obtained .

6. The method of claim 1, wherein, The step S5 specifically comprises the following steps: S51, updating the local control instruction sequence according to the current pose of the UAV is a new time; h is a control step.​​​​ S52, in every new moment Based on the global facility 3D semantic point cloud Filter out all component category labels that are equal to the current target component category. The points generate the current component's 3D point set. And based on the drone's current location Calculate the 3D point set of the UAV and the current target component. The closest distance between ; S53, Based on the nearest distance Construct coverage status indicator For the current target component category Perform semantic coverage determination of components, if Then the current target component category is considered to be... The semantic coverage requirement has been met, and the current target component category will be... Mark as completed inspection, then select the current target component category. Switch to component-level semantic inspection sequence The next target component category Proceed to step S4 until the component-level semantic inspection sequence is completed. All elements in the process are marked as having been inspected; if This indicates the current target component category. The semantic coverage has not yet been satisfied; the goal is to maintain [the target]. Proceed to step S3 to continue execution.

7. The method of claim 4, wherein, The existing open vocabulary detection segmentation framework that the open vocabulary visual perception network in step S31 can adopt has a core structure of an image encoder plus a text encoder plus a cross-modal matching locator head, and the input is a current frame image and a set of component categories The generated text prompt, the network will encode the text embedding and make a similarity match with the image features, thereby locating the corresponding region in the image, and outputting the number and the mask and category of the target component as the pixel-level segmentation result of the target component in the wide and high range of view of the current frame image .

8. The method of claim 5, wherein, The local path planning function in step S43 first constructs a three-dimensional occupancy grid map or a Euclidean distance field map in the world coordinate system, and then takes the current UAV position as the starting point, the target waypoint as the end point, and based on the obstacle set uses any feasible planning algorithm such as A*, D Lite*, or RRT on the map to obtain a collision-free discrete path, finally performs smoothing and dynamic feasibility processing on the discrete path to obtain a trajectory, and discretizes the trajectory into a control command sequence according to the flight control interface output.

9. The method of claim 6, wherein, In step S52, the current position of the UAV is determined The nearest distance between the UAV and the three-dimensional point set of the current target component is calculated The nearest distance between the UAV and the three-dimensional point set of the current target component is calculated As shown in the following formula: ， in It is a Euclidean norm function; Indicates the current time With the current target component category The number of related points; Represents the current target component's three-dimensional point set The point in the middle.

10. The method of claim 9, wherein, The coverage state indication quantity in step S53 The value of the coverage state indication quantity specifically includes: if there is a time point such that and , the coverage state indication quantity takes 1; otherwise, the coverage state indication quantity takes 0; wherein, The distance threshold for determining whether the semantic coverage is up to standard.

Images